Bowl-shaped imaging system

ABSTRACT

Technologies for utilizing a bowl-shaped image include a computing device to receive a first fisheye image capturing a first scene and a second fisheye image capturing a second scene overlapping with the first scene at an overlapping region. The computing device generates a combined image of the first fisheye image and the second fisheye image, performs object classification on a region of interest of at least one of the first fisheye image or the second fisheye image to classify an object within the region of interest, and generates a portion of a bowl-shaped image based on the combined image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/367,987, entitled “BOWL-SHAPED IMAGING SYSTEM,” which was filed onJun. 23, 2014, and is a national stage entry under 35 USC § 371 ofInternational Application No. PCT/US2013/076644, which was filed on Dec.19, 2013.

BACKGROUND

Vehicle-mounted camera systems such as rearview cameras have becomemainstays in luxury vehicles and even some lower-end vehicles. Dependingon the particular implementation, a vehicle camera system may be usedfor improved vision, automated parallel parking, and/or other purposes.For example, images captured by vehicle-mounted cameras may be displayedon a navigation system display screen of the vehicle providing thedriver with an unobstructed backward view (i.e., better than therearview mirror) or overhead view of the vehicle (i.e. showing thevehicle's surroundings).

Several cameras may be mounted to a vehicle to capture all of thesurroundings of the vehicle (i.e., the full 360 degrees). Suchvehicle-mounted camera systems may utilize wide-angle cameras such asthose having fisheye lenses (i.e., fisheye cameras) in order to minimizethe number of cameras necessary to capture the vehicle surroundings.Images of the surroundings of the vehicle are typically generated anddisplayed as inverse projections to the ground plane. Accordingly,objects located on the ground (e.g., parking lines) are typicallycorrectly projected; however, vertical objects such as other vehiclesand pedestrians have distorted projections because they are notcompletely located on the ground plane. Additionally, overlapping areasof neighboring cameras have different projections of a single real-worldpoint, which causes image ambiguity, duplicity, and/or invisibleobjects. Further, the distortions associated with fisheye images make itdifficult to identify and classify surrounding objects (e.g., vehicles,pedestrians, buildings, street signs, etc.).

BRIEF DESCRIPTION OF THE DRAWINGS

The concepts described herein are illustrated by way of example and notby way of limitation in the accompanying figures. For simplicity andclarity of illustration, elements illustrated in the figures are notnecessarily drawn to scale. Where considered appropriate, referencelabels have been repeated among the figures to indicate corresponding oranalogous elements.

FIG. 1 is a simplified block diagram of at least one embodiment of anin-vehicle computing system of a vehicle for displaying a physicalenvironment surrounding the vehicle;

FIG. 2 is a simplified illustration of at least one embodiment of thevehicle of FIG. 1;

FIG. 3 is a simplified block diagram of at least one embodiment of anenvironment established by the in-vehicle computing system of FIG. 1;

FIG. 4 is an illustration of at least one embodiment of a virtualbowl-shaped projection surface;

FIGS. 5A and 5B are simplified illustrations of a top view and a bottomview, respectively, of the virtual bowl-shaped projection surface ofFIG. 4;

FIG. 6 is a simplified illustration of a vertical cross-section of thevirtual bowl-shaped projection surface of FIG. 4;

FIG. 7 is a simplified flow diagram of at least one embodiment of amethod for displaying the physical environment surrounding the vehicleof FIG. 1;

FIG. 8 is a simplified illustration of at least one embodiment of aprojection of an object onto the virtual bowl-shaped projection surfaceof FIG. 4;

FIG. 9 is a simplified flow diagram of at least one embodiment of amethod for combining images generated by the in-vehicle computing systemof FIG. 1 utilizing image stitching;

FIG. 10 is a simplified illustrations of fisheye images captured by twoneighboring fisheye cameras;

FIG. 11 is a simplified illustration of stitched images from cameras ofthe vehicle of FIG. 1 projected onto the virtual bowl-shaped surface ofFIG. 4;

FIGS. 12 and 13 is a simplified flow diagram of at least one embodimentof a method for combining images generated by the in-vehicle computingsystem of FIG. 1 utilizing optical flow;

FIG. 14A is at least one embodiment of a fisheye image captured by thein-vehicle computing system of FIG. 1;

FIG. 14B is at least one embodiment of a generated optical flow outputbased on the captured fisheye image of FIG. 14A;

FIG. 15 is a simplified flow diagram of at least one embodiment of amethod for classifying objects in a fisheye image generated by thein-vehicle computing system of FIG. 1;

FIG. 16 is at least one embodiment of a fisheye image captured by afisheye camera of the in-vehicle computing system of FIG. 1;

FIG. 17 is a distorted rectilinear image generated from a projection ofthe fisheye image of FIG. 16;

FIG. 18 is a rectilinear image generated from the fisheye image of FIG.16 based on the method of FIG. 15; and

FIG. 19 is a simplified flow diagram of at least one embodiment of amethod for generating a bowl-shaped image by the in-vehicle computingsystem of FIG. 1;

DETAILED DESCRIPTION OF THE DRAWINGS

While the concepts of the present disclosure are susceptible to variousmodifications and alternative forms, specific embodiments thereof havebeen shown by way of example in the drawings and will be describedherein in detail. It should be understood, however, that there is nointent to limit the concepts of the present disclosure to the particularforms disclosed, but on the contrary, the intention is to cover allmodifications, equivalents, and alternatives consistent with the presentdisclosure and the appended claims.

References in the specification to “one embodiment,” “an embodiment,”“an illustrative embodiment,” etc., indicate that the embodimentdescribed may include a particular feature, structure, orcharacteristic, but every embodiment may or may not necessarily includethat particular feature, structure, or characteristic. Moreover, suchphrases are not necessarily referring to the same embodiment. Further,when a particular feature, structure, or characteristic is described inconnection with an embodiment, it is submitted that it is within theknowledge of one skilled in the art to effect such feature, structure,or characteristic in connection with other embodiments whether or notexplicitly described. Additionally, it should be appreciated that itemsincluded in a list in the form of “at least one A, B, and C” can mean(A); (B); (C): (A and B); (B and C); or (A, B, and C). Similarly, itemslisted in the form of “at least one of A, B, or C” can mean (A); (B);(C): (A and B); (B and C); or (A, B, and C).

The disclosed embodiments may be implemented, in some cases, inhardware, firmware, software, or any combination thereof. The disclosedembodiments may also be implemented as instructions carried by or storedon one or more transitory or non-transitory machine-readable (e.g.,computer-readable) storage medium, which may be read and executed by oneor more processors. A machine-readable storage medium may be embodied asany storage device, mechanism, or other physical structure for storingor transmitting information in a form readable by a machine (e.g., avolatile or non-volatile memory, a media disc, or other media device).

In the drawings, some structural or method features may be shown inspecific arrangements and/or orderings. However, it should beappreciated that such specific arrangements and/or orderings may not berequired. Rather, in some embodiments, such features may be arranged ina different manner and/or order than shown in the illustrative figures.Additionally, the inclusion of a structural or method feature in aparticular figure is not meant to imply that such feature is required inall embodiments and, in some embodiments, may not be included or may becombined with other features.

Referring now to FIG. 1, in the illustrative embodiment, a vehicle 100includes an in-vehicle computing system 102, which may be embodied asany type of computing system capable of performing the functionsdescribed herein. In the illustrative embodiment, the vehicle 100 isembodied as a wheeled passenger vehicle (e.g., a car, truck,truck-tractor, bus, etc.). However, it should be appreciated that, inother embodiments, the vehicle 100 may be embodied as another type ofvehicle (e.g., as a rail-driven trolley, an unmanned vehicle, or anothervehicle suited for application of the described techniques andmechanisms) or other moveable apparatus. As described herein, thein-vehicle computing system 102 is configured to capture fisheye images,combine the captured fisheye images, perform object classification onthe fisheye images, and display a bowl-shaped image based on thecombined fisheye images. In some embodiments, the in-vehicle computingsystem 102 may be embodied as, or form a portion of, an in-vehicleinfotainment system, a navigation system, and/or other vehicle-basedcomputing system. In other embodiments, the in-vehicle computing system102 may instead be embodied as a standalone computing device orcomputing system.

Further, in some embodiments, a remote computing device may becommunicatively coupled with the in-vehicle computing system 102 andconfigured to perform one or more of the functions described herein(e.g., remotely) and communicate the results to the in-vehicle computingsystem 102 via a network (e.g., a wired or wireless communicationnetwork). In such embodiments, the remote computing device may beembodied as any type of computing device capable of communicating withthe in-vehicle computing system 102 and performing the functionsdescribed herein (e.g., a server, desktop computer, laptop computer,tablet computer, notebook, netbook, Ultrabook™, a cellular phone,smartphone, personal digital assistant, mobile Internet device, Hybriddevice, and/or any other computing/communication device).

As shown in FIG. 1, the illustrative in-vehicle computing system 102includes a processor 110, an input/output (“I/O”) subsystem 112, amemory 114, a data storage 116, a display 118, a camera system 120, auser interface 122, and, in some embodiments, one or more peripheraldevices 124. Additionally, the camera system 120 includes one or morecameras 126 and the user interface 122 includes a virtual camera control128. Of course, the in-vehicle computing system 102 may include other oradditional components, such as those commonly found in a typicalcomputing device (e.g., communication circuitry, various input/outputdevices, and/or other components), in other embodiments. Additionally,in some embodiments, one or more of the illustrative components may beincorporated in, or otherwise from a portion of, another component. Forexample, the memory 114, or portions thereof, may be incorporated in theprocessor 110 in some embodiments.

The processor 110 may be embodied as any type of processor capable ofperforming the functions described herein. For example, the processormay be embodied as a single or multi-core processor(s), digital signalprocessor, microcontroller, or other processor or processing/controllingcircuit. Similarly, the memory 114 may be embodied as any type ofvolatile or non-volatile memory or data storage capable of performingthe functions described herein. In operation, the memory 114 may storevarious data and software used during operation of the in-vehiclecomputing system 102 such as operating systems, applications, programs,libraries, and drivers. The memory 114 is communicatively coupled to theprocessor 110 via the I/O subsystem 112, which may be embodied ascircuitry and/or components to facilitate input/output operations withthe processor 110, the memory 114, and other components of thein-vehicle computing system 102. For example, the I/O subsystem 112 maybe embodied as, or otherwise include, memory controller hubs,input/output control hubs, firmware devices, communication links (i.e.,point-to-point links, bus links, wires, cables, light guides, printedcircuit board traces, etc.) and/or other components and subsystems tofacilitate the input/output operations. In some embodiments, the I/Osubsystem 112 may form a portion of a system-on-a-chip (SoC) and beincorporated, along with the processor 110, the memory 114, and othercomponents of the in-vehicle computing system 102, on a singleintegrated circuit chip.

The data storage 116 may be embodied as any type of device or devicesconfigured for short-term or long-term storage of data such as, forexample, memory devices and circuits, memory cards, hard disk drives,solid-state drives, or other data storage devices. The data storage 116and/or the memory 114 may store various data during operation of thein-vehicle computing system 102 such as, for example, captured andprocessed images, object profiles and models, image filters, componentparameters, and/or other data useful in the operation of the in-vehiclecomputing system 102 as discussed below.

The display 118 of the in-vehicle computing system 102 may be embodiedas any one or more display screens on which information may be displayedto a user of the in-vehicle computing system 102 (e.g., a driver orpassenger of the vehicle 100). The display 118 may be embodied as, orotherwise use, any suitable display technology including, for example, aliquid crystal display (LCD), a light emitting diode (LED) display, acathode ray tube (CRT) display, a plasma display, a holographic or otherthree-dimensional (3D) display, and/or other display technology.Although only a single display 118 is illustrated in FIG. 1, it shouldbe appreciated that the in-vehicle computing system 102 may includemultiple displays or display screens on which the same or differentcontent may be displayed contemporaneously or sequentially with eachother. The display 118 may be embodied as a general purpose display onwhich various vehicle operating parameters and/or vehicle infotainmentinformation (e.g., radio station, temperature controls, etc.) aredisplayed. Alternatively, the display 118 may be embodied as a specificpurpose display for displaying images generated by the camera system 120as discussed below.

The camera system 120 includes one or more cameras 126, which may beused to capture images of a surrounding environment of the vehicle 100or, more particularly, a surrounding environment of the in-vehiclecomputing system 102. It should be appreciated that, in the illustrativeembodiment, the cameras 126 of the camera system 120 are suitably spacedto capture the full 360-degree surroundings of the vehicle 100 or asubstantial portion thereof. In some embodiments, the field of view ofeach camera 126 may overlap with one or more other cameras.Additionally, in some embodiments, the geometry associated with thepositioning of the cameras 126 relative to one another and/or relativeto the vehicle 100 is known or able to be determined.

Each of the cameras 126 may be embodied as any peripheral or integrateddevice suitable for capturing images, such as a still camera, a videocamera, a webcam, or other device capable of capturing video and/orimages. In the illustrative embodiment, each of the cameras 126 isembodied as a fisheye camera to facilitate capture of the complete, ornear complete, surroundings of the vehicle 100. Of course, in otherembodiments, other types of wide or narrow angled cameras may be useddepending on, for example, the type of vehicle 100, the type of cameras126 used, and/or other criteria. Additionally, the camera system 120 mayinclude two or more cameras 126 to capture the surroundings of thevehicle 100 depending on such various criteria. In the illustrativeembodiment, the camera system 120 includes four fisheye cameras 126(i.e., cameras having fisheye lenses) mounted to, or integrated with,the vehicle 100 to capture the full surroundings of the vehicle 100while minimizing the number of cameras 126 necessary to do so. Forexample, as shown in the illustrative embodiment of FIG. 2, a camera 126may be mounted to each of the four sides of the vehicle 100 (i.e., thefront, rear, driver's side, and passenger's side). Of course, in otherembodiments, the in-vehicle computing system 102 may utilize a differentnumber, type, and/or relative positioning of the cameras 126. Forexample, in another embodiment, the cameras 126 may be positioned inother locations about the vehicle 100 (e.g., at the corners of thevehicle 100).

The user interface 122 permits a user of the in-vehicle computing system102 to interact with the in-vehicle computing system 102. For example,as discussed below, the user may interact with the in-vehicle computingsystem 102 to display the surroundings of the vehicle 100 on the display118. As such, in some embodiments, the user interface 122 includes oneor more virtual and/or physical buttons, knobs, switches, keypads,touchscreens, and/or other mechanisms to permit I/O functionality. Forexample, in some embodiments, the user interface 122 may be integratedwith a navigation or infotainment system of the vehicle 100. Asdiscussed above, the user interface 122 includes a virtual cameracontrol 128 in the illustrative embodiment. The virtual camera control128 permits a user of the in-vehicle computing system 102 to modify thefield of view of a virtual camera 202 (see FIG. 2) of the in-vehiclecomputing system 102 (e.g., by “moving” the virtual camera). Asdiscussed in detail below, the virtual camera 202 “captures” the imageto be rendered on the display 118, which may correspond with a singleimage captured by one of the cameras 126 or a combination of imagescaptured by multiple cameras 126 (e.g., adjacent cameras 126).Accordingly, by rotating or otherwise moving the virtual camera 202 andthereby changing the field of view of the virtual camera 202, thedisplayed image may be adjusted by the user of the in-vehicle computingsystem 102. As discussed in further detail below, a bowl-shaped image isgenerated based on images of the surroundings of the vehicle 100captured by the cameras 126 and processed by the in-vehicle computingsystem 102. As such, in some embodiments, the bowl-shaped image may berendered on the display 118 based on the field of view of the virtualcamera 202, which may include a combination of field of views frommultiple cameras 126. For example, the bowl-shaped image may bedisplayed based on a perspective from which the bowl-shaped image is“captured” by the virtual camera 202 (e.g., from the top/bottom, fromfar/near, from inside/outside the bowl-shaped image, etc.).

In some embodiments, the in-vehicle computing system 102 may alsoinclude one or more peripheral devices 124. The peripheral devices 124may include any number of additional peripheral or interface devices,such as speakers, microphones, additional storage devices, and so forth.The particular devices included in the peripheral devices 124 may dependon, for example, the type and/or intended use of the in-vehiclecomputing system 102 (e.g., whether the in-vehicle computing system 102is a stand-alone system or incorporated into a larger in-vehicleinfotainment system).

Referring now to FIG. 3, in use, the illustrative in-vehicle computingsystem 102 establishes an environment 300 for bowl-shaped imaging of thesurroundings of the vehicle 100. As discussed below, the in-vehiclecomputing system 102 combines fisheye images captured by the cameras 126in a way that minimizes, or otherwise reduces, image ambiguity,duplicity, and/or invisible objects. The in-vehicle computing system 102may further perform object classification on the fisheye images anddisplays a bowl-shaped image based on the combined fisheye images toprovide improved perception of the vehicle 100 surroundings to theoccupants of the vehicle 100 (e.g., the driver). The illustrativeenvironment 300 of the in-vehicle computing system 102 includes an imagecapturing module 302, a combined image generation module 304, an objectclassification module 306, a bowl generation module 308, a displaymodule 310, and a camera control module 312. Additionally, the combinedimage generation module 304 includes a stitching module 314 and anoptical flow module 316. As shown in the illustrative embodiment, thestitching module 314 includes a feature detection module 318, a featurematching module 320, a seam determination module 322, and an imagesynthesis module 324; and the optical flow module 316 includes anoverlap determination module 326, a movement determination module 328,and an image generation module 330. Further, the object classificationmodule 306 includes a calibration module 332, a region of interest (ROI)determination module 334, a rectilinear projection module 336, and anobject identification module 338; and the bowl generation module 308includes a bowl geometry module 340, a bowl synthesis module 342, and abowl warping module 344. Each of the modules of the environment 300 maybe embodied as hardware, software, firmware, or a combination thereof.Additionally, in some embodiments, one or more of the illustrativemodules may form a portion of another module.

The image capturing module 302 controls the cameras 126 of the camerasystem 120 to capture images/video of scenes within the field of view ofeach of the cameras 126. Depending on the particular embodiment, theimage capturing module 302 may instruct each camera 126 to captureimages continuously (e.g., as a video stream), periodically, in responseto temporal or conditional inputs, as a combination thereof, or based onanother scheme. For example, the cameras 126 may capture images when thetransmission of the vehicle 100 is in reverse mode or when the vehicle100 is moving below a certain threshold speed.

As discussed above, the camera system 120 includes a plurality ofcameras 126 strategically placed to capture the surroundings of thevehicle 100. Each of the cameras 126 generates an independent image ofthe scene within its field of view. As such, the combined imagegeneration module 304 combines two or more images generated by thecameras 126 to generate combined images. For example, in theillustrative embodiment, the combined image generation module 304generates a combined image based on two images captured by neighboringcameras 126 (i.e., cameras 126 most adjacent one another). Forconvenience, such images may be referred to herein as “neighboringimages.” In some embodiments, the two or more images used by thecombined image generation module 304 to generate the combined image maybe based on, for example, the position of the virtual camera 202 (i.e.,based on the surrounding area that the user desires to view). In otherembodiments, multiple combined images may be generated based on each ofimages generated by the plurality of cameras 126.

It should be appreciated that, in the illustrative embodiment, thecamera system 120 is arranged such that the fields of view of theneighboring cameras 126 have an overlapping region. As such, when theimages captured by those neighboring cameras 126 are projected onto avirtual bowl-shaped surface, the bowl-shaped surface oftentimes hasdifferent projections of a single real-world point, which may causeimage ambiguity, duplicity, and/or invisible objects in the surroundingsof the vehicle 100. In combining images captured by neighboring cameras126, the combined image generation module 304 minimizes, or otherwisereduces, those projection problems. To do so, the combined imagegeneration module 304 may utilize, for example, image stitching, opticalflow mechanisms, image blending, and/or other suitable technologies asdiscussed below. In some embodiments, the combined image generationmodule 304 generates a “panoramic” image based on the pairwisecombination of images (i.e., by combining images generated byneighboring cameras 126).

The stitching module 314 combines captured images (e.g., fisheye images)and/or their projections onto a virtual bowl-shaped surface based onimage stitching. For example, in the illustrative embodiment, thestitching module 314 utilizes image stitching to generate a combinedimage based on images captured by neighboring cameras 126. As discussedabove, the geometry of the camera system 120 causes the fields of viewof neighboring cameras 126 to overlap which, in turn, causes the imagescaptured by the neighboring cameras 126 to have scenes that overlap atan overlapping region. In other words, in the overlapping region, thecaptured images include depictions of the same real-world points of theenvironment surrounding the vehicle 100 and therefore include the sameimage features, which may or may not directly overlap each other. Toperform the image stitching, the stitching module 314 includes thefeature detection module 318, the feature matching module 320, the seamdetermination module 322, and the image synthesis module 324.

The feature detection module 318 of the stitching module 314 performsfeature detection on images captured by the neighboring cameras 126. Itshould be appreciated that the feature detection module 318 may utilizeany suitable feature detection techniques, algorithms, and/or mechanismsto detect/identify features (e.g., interest points such as corners,edges, blobs, etc.) of an image. For example, the feature detectionmodule 318 may utilize common feature detectors/descriptors such asSpeeded Up Robust Features (SURF), Scale-Invariant Feature Transform(SIFT), Multi-Scale Oriented Patches (MOPS), Canny, Harris, and/or Sobelfilters to detect and describe features of an image.

The feature matching module 320 performs feature matching on thedetected features of the images captured by the neighboring cameras 126to identify which features (e.g., the specific pixels) of the imageswithin the overlapping region correspond with one another. Of course,the feature matching module 320 may utilize any suitable featurematching techniques, algorithms, and/or mechanisms for identifyingfeatures of two images (i.e., a stereo image pair) that correspond withone another. For example, the feature matching module 320 may utilizeone or more of the detectors/descriptions discussed above in conjunctionwith a matching algorithm or technique such as squared differences(SSD), fixed threshold, nearest neighbor, and/or Random Sample Consensus(RANSAC). Additionally or alternatively, in some embodiments, thefeature matching module 320 may utilize the predetermined geometry ofthe camera system 120 to employ deterministic techniques for identifyingmatching features in the overlapping region.

Based on the matched features, the seam determination module 322identifies a seam on each of the neighboring images (i.e., a first seamon a first image and a second seam on a second image) in the overlappingregion. That is, the seam determination module 322 determines whichfeatures (e.g., the specific image location or pixel) of the first imagematch a corresponding feature of the second image and “connects” thosefeatures to determine a virtual seam for the first image (e.g., bydrawing a line segment between each adjacent matched feature in thefirst image in a “connect-the-dots” manner). Likewise, the seamdetermination module 322 determines a seam for the second image based onthe matched features of that image. Based on the seams of the first andsecond images, the seam determination module 322 determines a third seamat which to combine the first and second images. It should beappreciated that the seam determination module 322 may determine thelocation of the third seam using any suitable technique (e.g., seamaveraging). For example, in the illustrative embodiment, the seamdetermination module 322 projects the first and second images onto avirtual bowl-shaped projection surface as described below (e.g., via thebowl generation module 308). As such, the determined seams are alsoprojected onto the virtual bowl-shaped surface and, based on thoseprojected seams, the seam determination module 322 determines the thirdseam at which to combine the first and second images or, moreparticularly, the projections of the first and second images.

The image synthesis module 324 generates a combined image of theneighboring images (or the corresponding image projections) by combiningthe images/projections at the third seam determined by the seamdetermination module 322. Additionally, the image synthesis module 324may interpolate and/or extrapolate the image pixels between the firstand second seams of the neighboring images or image projections togenerate a “smooth” image and remove image artifacts and/or mayotherwise filter the overlapping region of the images. Further, ingenerating the combined image, the image synthesis module 324 maydiscard or deemphasize portions of the first image or projection beyondthe first seam and portions of the second image or projection beyond thesecond seam (i.e., the portions within the overlapping image region butoutside the region defined between the first and second seam). Thefunctionality of the stitching module 314 is further discussed below inregard to method 900 of FIG. 9.

The optical flow module 316 of the combined image generation module 304combines captured images (e.g., fisheye images) based on optical flow.In the illustrative embodiment, the generated combined image includesthe non-overlapping portions of the neighboring images and an imagegenerated based on the overlapping image region. As discussed above, theoptical flow module 316 includes the overlap determination module 326,the movement determination module 328, and the image generation module330. It should be appreciated that, depending on the particularembodiment, the in-vehicle computing system 102 may combine imagesutilizing image stitching, optical flow, and/or other image combiningmechanisms.

The overlap determination module 326 determines an overlapping region ofneighboring images. It should be appreciated that the overlapdetermination module 326 may identify overlapping image regions usingany suitable techniques, algorithms, or mechanisms. For example, asdiscussed above, the geometry of the camera system 120 is known ordeterminable. Accordingly, in some embodiments, the overlapping regionmay be determined based on such geometry. In other embodiments, theoverlapping region may be determined based on, for example, the featuredetection and matching discussed above.

The movement determination module 328 determines movement of objectslocated in the overlapping image region relative to the cameras 126 thatcaptured the neighboring images. As discussed in detail below, in doingso, the movement determination module 328 may use multiple imagescaptured by a camera 126 to determine the time derivative of theintensity of the overlapping image region for each of the neighboringcameras 126. Additionally, the movement determination module 328 maydetermine the space derivative of the intensity of the overlapping imageregion for the neighboring images. Based on the determined relativemovement of objects in the field of view of the neighboring cameras 126within the overlapping region, the movement determination module 328identifies objects moving relative to the vehicle 100.

The image generation module 330 generates an image of the overlappingimage region based on the identified moving objects. As discussed above,the generated image may be combined with the non-overlapping portions ofthe neighboring images to result in a combined image of the neighboringimages. In generating an image of the overlapping image region, theimage generation module 330 may generate an image that includes theidentified moving objects from one or both of the neighboring images,blend or otherwise filter the overlapping image region, and/or positionthe moving objects in estimated positions. The functionality of theoptical flow module 316 is further discussed below in regard to method1200 of FIGS. 12 and 13.

The object classification module 306 detects and classifies objectspresent in fisheye images captured by the cameras 126 of the vehicle100. For example, the object classification module 306 may utilizeobject classification techniques to determine that a pedestrian iswalking in a blind spot of the vehicle 100 and, in some embodiments, mayalert the driver of the vehicle 100. Although object classification isdescribed herein with respect to fisheye images, it should beappreciated that the description may be applied to other types of images(e.g., images generated by cameras having a different type of wide-anglelens). As indicated above, the object classification module 306 includesthe calibration module 332, the region of interest determination module334, the rectilinear projection module 336, and the objectidentification module 338.

The calibration module 332 is configured to calibrate the fisheyecameras 126 for accurate image capturing and extraction of metricinformation (e.g., distances between objects and/or image features)necessary for effective object classification. It should be appreciatedthat the calibration module 332 may employ any suitable techniques,algorithms, and/or mechanisms for doing so. For example, depending onthe particular embodiment, the calibration module 332 may calibrate thecameras 126 using techniques directed to the particular sensor/lens type(e.g., a specific type of fisheye lens) and/or using techniquesgenerally applicable to a broader category of sensors/lenses (e.g., allsensors/lenses). Additionally, it should be appreciated that thecalibration module 332 may perform such calibration at a particularpoint in time (e.g., during a boot process of the in-vehicle computingsystem 102) or periodically over time.

The region-of-interest determination module 334 determines a region ofinterest in a captured fisheye image to which the object classificationis to be focused. As discussed in detail below, performing objectclassification on an entire image is generally computationallyintensive. Additionally, the distortion inherent in fisheye images(e.g., fisheye distortion) typically makes performing objectclassification on fisheye images quite difficult, because identicalobjects located in different parts of a fisheye image may appear to havedifferent shapes (e.g., due to a lack of translation, scale, androtation invariance in fisheye images) and object profiles are typicallydefined in undistorted rectilinear space. Although conversion of anentire fisheye image to a rectilinear image reduces some distortion, ittypically, nonetheless, results in significant image distortion (e.g.,perspective distortion) due to the image projections employed in theconversion. By focusing on a sub-set of the entire image (i.e., a“region-of-interest”), the computation complexity and/or imagedistortion can be reduced. The region of interest determination module334 may use any suitable techniques for identifying the region ofinterest (e.g., optical flow or object detection on the original fisheyeimage or intermediate image, laser scanning, radar, a photonic mixingdevice, etc.).

The rectilinear projection module 336 converts the region of interest ofthe fisheye image into a rectilinear image using image projection. Inparticular, the rectilinear projection module 336 projects theidentified region of interest onto a virtual hemispherical displaysurface to generate a hemispherical image region (i.e., an image“wrapped” around a virtual hemispherical surface). Further, therectilinear projection module 336 projects the hemispherical imageregion onto a plane perpendicular to a point in the hemispherical imageregion (e.g., the center point) to generate a rectilinear image. Theresulting rectilinear image tends to be significantly less distortedthan the rectilinear image resulting from projection of the entirefisheye image to a plane. In other embodiments, the rectilinearprojection module 336 may convert the region of interest into arectilinear image using other techniques.

The object identification module 338 performs object classification onthe rectilinear image to detect and/or classify objects in the vicinitysurrounding the vehicle 100. For example, as discussed below, the objectidentification module 338 may compare objects identified in therectilinear image to predefined object profiles stored on the memory 114or data storage 116 of the in-vehicle computing system 102 (or storedremotely) to determine whether there is a match. After classifying anobject in the rectilinear image, the object classification module 306may determine the location of the fisheye image from which theclassified object was projected (e.g., via back projection).Additionally, in some embodiments, the object classification module 306may alert a user of the in-vehicle computing system 102 of theclassification (e.g., by highlighting the classified object or via anaudible alert message). The functionality of the object classificationmodule 306 is further discussed below in regard to method 1500 of FIG.15.

Referring now to FIGS. 3-6, the bowl generation module 308 generates avirtual bowl-shaped projection surface 402 (see FIG. 4) on which imagescaptured by the cameras 126 of the camera system 120, or processedversions thereof (e.g., combined images), may be projected for viewingby the user of the in-vehicle computing system 102 (e.g., the driver ofthe vehicle 100). In most in-vehicle imaging systems, visualizations ofimages captured by vehicle-mounted cameras result in significantdistortion of objects that are not located on a ground plane 404 (seeFIG. 4) upon which the vehicle rests. As described herein, the virtualbowl-shaped projection surface has specifically defined geometriccharacteristics such as, for example, a smoothness of the virtualbowl-shaped projection surface 402, a smooth transition between thevirtual bowl-shaped projection surface 402 and the ground plane 404, andflatness of the bowl-shaped projection surface 402 at the ground plane404. Accordingly, by projecting the images to the virtual bowl-shapedprojection surface 402, one or more advantages may be obtained such as,for example, the reduction of object distortion and/or the possibilityof seeing objects high above the ground plane 404, which may be hiddenor otherwise non-viewable with a traditional planar virtual projectionsurface. As indicated above, in the illustrative embodiment, the bowlgeneration module 308 includes the bowl geometry module 340, the bowlsynthesis module 342, and the bowl warping module 344.

The bowl geometry module 340 determines the desired geometry of thevirtual bowl-shaped projection surface 402 based on various parametersand constructs each horizontal slice (see, for example, slice 406 ofFIG. 4) of the virtual bowl-shaped projection surface 402. In doing so,the bowl geometry module 340 determines (e.g., via retrieval)information related to the size of the vehicle 100 (e.g., length, width,height, etc.) and retrieves various parameters (e.g., bowl height,horizontal offset, adaptive ratios, etc.). As indicated below, the widthof each horizontal slice may vary depending on the particularembodiment. Further, in some embodiments, the horizontal slices may havenon-uniform (i.e., varying) widths.

It should be appreciated that, in the illustrative embodiment, thevirtual bowl-shaped projection surface 402 is shaped such that thevehicle 100, or a virtual representation thereof, is positioned insidethe bowl-shaped projection surface 402 on the ground plane 404. Asdescribed in detail below, based on that information, the bowl geometrymodule 340 determines the major and minor semi-axes of a vertical slice408 (see FIG. 4) of the bowl-shaped projection surface 402 along thelength of the vehicle 100. Additionally, the bowl geometry module 340determines, for each horizontal slice of the virtual bowl-shapedprojection surface 402 (e.g., from bottom 410 to top 412), the majorsemi-axis of the corresponding horizontal slice based on the major andminor semi-axes of the vertical slice 408 and the minor semi-axis of thecorresponding horizontal slice based on an adaptive ratio between themajor and minor semi-axes of the corresponding horizontal slice. Asdescribed below, in the illustrative embodiment, the adaptive ratiochanges linearly and is approximately unity (i.e., 1.0) at the top 412of the bowl-shaped projection surface 402 and equal to the length of thecar divided by the width of the car

$\left( {{i.e.},\frac{carLength}{carWidth}} \right)$

at the bottom 410 of the bowl-shaped projection surface 402. As shown inFIG. 5A, from a top view 502 of the bowl-shaped projection surface 402looking along an axis 504 of the bowl-shaped projection surface 402, thetop-most horizontal slice 506 of the bowl-shaped projection surface 402has a circular geometric shape and a largest slice 508 of thebowl-shaped projection surface 402 is toward the vertical middle of thebowl-shaped projection surface 402. Further, as shown in FIG. 5B, from abottom view 510 of the bowl-shaped projection surface 402 looking alongthe axis 504, the bottom-most horizontal slice 512 of the bowl-shapedprojection surface 402 has a non-circular elliptical geometric shape(i.e., the bottom-most horizontal slice 512 defines an ellipse for whichits major axis does not equal its minor axis).

The bowl synthesis module 342 generates the virtual bowl-shapedprojection surface 402 based on a combination of the horizontal slicesgenerated by the bowl geometry module 340. For example, the bowlsynthesis module 342 may stack the horizontal slices of the bowl-shapedprojection surface 402 and merge those slices together to form theresultant bowl-shaped projection surface 402. Additionally, the bowlsynthesis module 342 may generate and merge a bottom 410 of the bowl,which is a flat ground plane 404 (or partial-plane, more specifically).It should be appreciated that the top 412 of the bowl-shaped projectionsurface 402 is partially open and portions of the environment above thevehicle 100 may be projected on a top portion of the bowl-shapedprojection surface 402.

The bowl warping module 344 is configured to filter or otherwise smootha section of the generated virtual bowl-shaped projection surface 402adjacent the ground plane 404 (e.g., the bottom half, the bottom third,the bottom quarter, the bottom eighth, etc. depending on the particularembodiment). As shown in FIG. 6 by a portion of a vertical cross section602 of the bowl-shaped projection surface 402 (e.g., taken perpendicularto the vertical slice 408), the bowl warping module 344 modifies theshape of the constructed bowl-shaped projection surface 402 such that abottom portion 604 of the bowl-shaped projection surface 402 approachesthe bottom 410 of the bowl-shaped projection surface 402 (i.e., theground plane 404) at a slower rate than a top portion 606 of thebowl-shaped projection surface 402 approaches the top 412 of thebowl-shaped projection surface 402. As such, an image projected on thebowl-shaped projection surface 402 (referred to herein as “a bowl-shapedimage”) will include those features described above with regard to thebowl-shaped projection surface 402 (i.e., the bowl-shaped image willinclude a bottom portion having a derivative with an absolute value thatis less than the absolute value of the derivative of the top portion.Further, the bottom portion of the bowl-shaped projection surface 402merges with the ground plane 404 of the bowl-shaped projection surface402 such that the derivative of the surface at any place is definite(i.e., there exist no singularities). It should be appreciated that thebowl warping module 344 may use any suitable image filters, algorithms,techniques, and/or mechanisms to do so.

Referring back to FIG. 3, the display module 310 renders images on thedisplay 118 for a user of the in-vehicle computing system 102 to view.For example, the display module 310 may “project” one or more fisheyeimages, combined images, and/or other types of images onto the virtualbowl-shaped projection surface 402 generated by the bowl generationmodule 308 to create a bowl-shaped image and display the bowl-shapedimage, or a portion thereof, on the display 118. Of course, it should beappreciated that, in some embodiments, the generated images or combinedimages are not “projected” onto the bowl-shaped projection surface 402in the traditional sense; rather, the generated images or combinedimages may be modified to have a bowl shape, including the featuresdiscussed above, as if the image(s) were projected onto the bowl-shapedprojection surface 402.

As indicated above, in some embodiments, the display module 310 mayrender only the portions of the bowl-shaped image within the field ofview of a virtual camera 202 of the in-vehicle computing system 102.Accordingly, although the bowl-shaped image may be embodied as an image“projected” onto a three-dimensional surface, the display 118 may rendera two-dimensional perspective of the bowl-shaped image in someembodiments. The camera control module 312 determines the perspective ofthe virtual camera 202 and, in conjunction with the virtual cameracontrol 128, permits a user of the in-vehicle computing system 102 tomodify the field of view of the virtual camera 202 of the in-vehiclecomputing system 102.

Referring now to FIG. 7, in use, the in-vehicle computing system 102 mayexecute a method 700 for displaying the physical environment surroundingthe vehicle 100 or, more particularly, the in-vehicle computing system102. As discussed above, the methods described herein are generallydescribed with respect to fisheye cameras and fisheye images generatedby such cameras; however, the methods are also be applicable to othersuitable types of cameras and/or images. The illustrative method 700begins with block 702 in which the in-vehicle computing system 102determines whether to display the surrounding physical environment. Ifso, the in-vehicle computing system 102 receives fisheye images from thecameras 126 in block 704. As discussed above, in the illustrativeembodiment (see FIG. 2), the camera system 120 includes four fisheyecameras. Accordingly, in such an embodiment, the in-vehicle computingsystem 102 receives fisheye images from each of the four cameras 126associated with the same capture time (i.e., captured at the same timeor approximately the same time). Of course, in other embodiments, thecamera system 120 may include a different number of cameras 126 andtherefore a different number of fisheye images may be received orretrieved in block 704. Additionally, in some embodiments, one or moreof the images may be stored on the memory 114 or data storage 116 of thein-vehicle computing system 102 (e.g., for use with optical flowalgorithms discussed below).

In block 706, the in-vehicle computing system 102 combines two or moreof the images received from the cameras 126 to generate acombined/composite image. It should be appreciated that the in-vehiclecomputing system 102 may use any suitable techniques or mechanisms fordoing so. For example, in a relatively rudimentary embodiment, thein-vehicle computing system 102 may simply append the images to oneanother to generate the combined image. However, as discussed below, theillustrative embodiments may utilize a more complex technique, such asstitching and/or optical flow, to combine the images to reduceinaccuracies or ambiguity in the generated combined image and/or thecorresponding projection.

As discussed above, the in-vehicle computing system 102 generates avirtual bowl-shaped projection surface 402 to which the captured imagesand/or combined images may be projected for the convenience and improvedvision of the driver of the vehicle 100. Additionally, neighboringcameras 126 have fields of view that overlap, which causes the imagesgenerated by the neighboring cameras 126 to have features that match(i.e., are duplicated) in the overlapping image region. Such duplicationof features can result in ambiguity of such features in the combinedimage and/or corresponding projections if not addressed. For example, asshown in FIG. 8, two neighboring cameras 802, 804 project (e.g., viaback projection) the same real-world point 806 onto two different points808, 810 of a bowl-shaped projection surface 402 (i.e., unless thatreal-world point 806 is coincident with the bowl-shaped projectionsurface 402). In particular, the real-world point 806 as captured by thefirst camera 802 is projected onto the bowl-shaped projection surface402 at a first point 808 and the real-world point 806 as captured by thesecond camera 802 is projected onto the bowl-shaped surface at a secondpoint 810. As such, when combining images, the in-vehicle computingsystem 102 may address the issues with respect to image ambiguity,duplicity, and/or invisible objects due to overlapping image regions.For example, in the illustrative embodiment, the in-vehicle computingsystem 102 may combine two adjacent images by utilizing a stitchingalgorithm in block 708 or by the utilizing an optical flow algorithm inblock 710 of the method 700 of FIG. 7. Of course, in some embodiments, acombination of techniques may be employed to generate the combinedimage.

To combine neighboring images using a stitching algorithm, thein-vehicle computing system 102 may execute a method 900 as shown inFIG. 9. The illustrative method 900 begins with block 902 in which thein-vehicle computing system 102 determines whether to stitch neighboringfisheye images. If so, in block 904, the in-vehicle computing system 102determines a fundamental matrix for the neighboring images based on thegeometry of the camera system 120. It should be appreciated that afundamental matrix relates image coordinates of two images capturing thesame scene to one another by virtue of mappings between correspondingfeatures of the images. Because the geometry and coordinate system ofthe camera system 120 is known, the fundamental matrix is also known oreasily determinable (i.e., without performing image feature detection ormatching). Alternatively, the fundamental matrix may be determined basedon matched features of the two images.

In block 906, the in-vehicle computing system 102 performs featuredetection on each of the neighboring fisheye images to identify features(i.e., interest points) of the images. As discussed above, thein-vehicle computing system 102 may use any suitable detectors for doingso (e.g., SURF, SIFT, MOPS, etc.). In block 908, the in-vehiclecomputing system 102 performs feature matching on the detected featuresof the neighboring fisheye images to identify which features of theimages correspond with one another (e.g., using RANSAC or anothermatching algorithm). In some embodiments, in doing so, the in-vehiclecomputing system 102 filters the feature match candidates based on thedetermined fundamental matrix in block 910 (e.g., to reduce thecomputational overhead associated with comparing the detected features).

Additionally or alternatively, in performing feature matching, thein-vehicle computing system 102 may minimize the Samson's distance ofthe feature match candidates to identify the appropriate featurecorrespondences between the overlapping images in block 912. Morespecifically, the in-vehicle computing system 102 may minimize theexpression,

${\sum\limits_{i}\frac{\left( {x_{i}^{\prime \; T}{Fx}_{i}} \right)^{2}}{\left( {Fx}_{i} \right)_{1}^{2} + \left( {Fx}_{i} \right)_{2}^{2} + \left( {Fx}_{i}^{\prime} \right)_{1}^{2} + \left( {Fx}_{i}^{\prime} \right)_{2}^{2}}},$

where x₁ and x₂ are the two-dimensional coordinates of a pixelassociated with a matched feature of the first fisheye image of theoverlapping images, x₁′ and x₂′ are the coordinates of a pixelassociated with a matched feature of the second fisheye image of theoverlapping images, F is the fundamental matrix, and i is an index ofpixels associated with the matched features. In other embodiments, thein-vehicle computing system 102 may perform feature matching using othersuitable techniques (e.g., using the determined fundamental matrixwithout feature matching algorithms, using feature matching algorithmswithout a priori knowledge of the fundamental matrix from the camerasystem 120 geometry, etc.).

In block 914, the in-vehicle computing system 102 determines the seamsof the neighboring fisheye images in the overlapping image region basedon the matched features. For example and referring now to FIGS. 10 and11, the in-vehicle computing system 102 may identify image features(e.g., pixels or groups of pixels) as interest points for eachneighboring image (i.e., for each of a first fisheye image 1002 and asecond fisheye image 1004). The subset of the detected features of thefirst image 1002 that match a corresponding features of the second image1004 may be virtually connected (e.g., in a “connect-the-dots” manner orotherwise) to generate a seam 1006 composed of the matched features ofthe first image 1002. Similarly, the detected features of the secondimage 1004 that match corresponding features of the first image 1002 mayalso establish a seam 1008.

Referring back to FIG. 9, in block 916, the in-vehicle computing system102 generates a combined image of the neighboring fisheye images. Asdiscussed above, in doing so, the in-vehicle computing system 102 mayproject the neighboring fisheye images (i.e., the images 1002, 1004 ontothe virtual bowl-shaped projection surface 402 in block 918. It shouldbe appreciated that the method 1900 as shown in FIG. 19 and describedbelow may be utilized or otherwise performed in conjunction with themethod 900 (e.g., at least in part) to perform such projection onto thebowl-shaped projection surface 402. In block 920, the in-vehiclecomputing system 102 determines a third seam 1102 at which theprojections of the first image 1002 and the second image 1004 are to bejoined. It should be appreciated that the in-vehicle computing system102 may determine the location of the third seam 1102 using any suitabletechniques, algorithms, or mechanisms (see, e.g., FIG. 11). For example,in one embodiment, the in-vehicle computing system 102 may determine thethird seam 1102 as an average seam between the seams 1006, 1008 (e.g.,an arithmetic mean of locations of the corresponding matched features ofthe seams 1006, 1008 on the three-dimensional bowl-shaped projectionsurface 402). In another embodiment, the location of the third seam 1102may be chosen, for example, as the location able to preserve the mostdata of the images 1002, 1004 or the corresponding projections. In block922, the in-vehicle computing system 102 may interpolate, extrapolate,and/or otherwise filter the image pixels between the first and secondseams 1006, 1008 of the projected images to generate a projected imagerelatively free of image artifacts in the overlapping image region. Insome embodiments, the in-vehicle computing system 102 may additionallydeemphasize portions of the projected images (e.g., portions beyond theseams 1006, 1008 in the corresponding images 1002, 1004). For example,the in-vehicle computing system 102 may crop or blur such portions ofthe image projections. In another embodiment, the in-vehicle computingsystem 102 may determine the third seam and filter the region betweenthe original seams in two-dimensional space prior to generate a combinedimage prior to projecting the combined image to the bowl-shapedprojection surface 402. Further, as shown in FIG. 11, it should beappreciated that seams 1104 may be generated between each other pair ofneighboring cameras 126 in a manner similar to that described above.

To combine neighboring images using an optical flow algorithm, thein-vehicle computing system 102 may execute a method 1200 as shown inFIGS. 12-13. The illustrative method 1200 begins with block 1202 inwhich the in-vehicle computing system 102 determines whether to utilizeoptical flow. It should be appreciated that the in-vehicle computingsystem 102 may utilize optical flow to determine how to project anoverlapping image region of neighboring fisheye images to the virtualbowl-shaped projection surface 402 based on whether any objects aredetermined to be moving within the overlapping fields of view. If thein-vehicle computing system 102 determines to utilize optical flow, inblock 1204, the in-vehicle computing system 102 determines/identifiesthe overlapping region of neighboring fisheye images to which to applyoptical flow. For example, in block 1206, the in-vehicle computingsystem 102 may determine the overlapping region of the neighboringfisheye images based on the predetermined geometry of the camera system120. That is, the in-vehicle computing system 102 may be configured toidentify the overlapping region of the neighboring fisheye images basedon the overlapping fields of view of the corresponding neighboringcameras 126 and the known geometric relationship among the cameras 126.Additionally or alternatively, in block 1208, the in-vehicle computingsystem 102 may perform feature detection and feature matching on theneighboring fisheye images to determine the overlapping region asdiscussed above with regard to FIG. 9.

In block 1210, the in-vehicle computing system 102 determines therelative movement of objects in the identified overlapping image region.That is, the in-vehicle computing system 102 determines the extent towhich, if at all, objects captured in the overlapping image region aremoving with respect to the vehicle 100. In doing so, the in-vehiclecomputing system 102 may utilize image filters (e.g., edge detectors,gradient filters, etc.) or other suitable techniques for determining therelative movement of objects in the overlapping region. For example, inblock 1212, the in-vehicle computing system 102 may apply a gradientconstraint equation. As such, in block 1214, the in-vehicle computingsystem 102 may determine the space derivative of the intensity of thepixels in the overlapping image region (e.g., for one or both of theneighboring images). Additionally, in block 1216, the in-vehiclecomputing system 102 may determine the time derivative of the intensityof the pixels in the overlapping image region.

In some embodiments, a gradient constraint equation,∇I(x,t)u+I_(t)(x,t)=0, may be applied to the overlapping image region toascertain movement of objects within the overlapping image region, whereI(x,t) is the pixel intensity, ∇I(x,t) is the space derivative of thepixel intensity, u is a two-dimensional velocity of the plurality offisheye images, and I_(t)(x,t) is the time derivative of the pixelintensity. More specifically, u is the two-dimensional velocity of theunderlying pixels determined by comparing two fisheye images taken bythe same fisheye camera 126 (e.g., with a short duration betweencaptures). It should be appreciated that a multitude of optical flowalgorithms may utilize such a gradient constraint equation as a basis,and any of those algorithms may be employed herein. Further, in someembodiments, optical flow algorithms having a different physicalconstraint may be employed by the in-vehicle computing system 102. Asshown in the illustrative embodiment of FIGS. 14A and 14B, an opticalflow algorithm may output an optical flow image 1404 based on an inputimage 1402 thereby indicating the relative movement of objects in theimage. It should be appreciated that the illustrative input image 1402of the optical flow algorithm shown in FIG. 14A is a fisheye imageincluding a scene with a non-moving background and a moving person. Assuch, the optical flow image 1404 (i.e., the output image of the opticalflow algorithm) clearly distinguishes the moving person from thenon-moving background with different colors for further analysis.Specifically, the utilized optical flow algorithm identifies movingobjects with darker shades and non-moving objects with lighter shades.In the illustrative embodiment, such an algorithm is applied to theoverlapping image region to generate a similar corresponding opticalflow output image.

In block 1218 (see FIG. 13), the in-vehicle computing system 102identifies any objects moving relative to the vehicle 100 based on therelative movement of objects in the overlapping image region (e.g.,based on the optical flow output image). It should be appreciated thatsuch a determination may depend, for example, on the particular opticalflow algorithm employed. For example, one particular optical flowalgorithm may output an image in grayscale in which pixels of theprocessed image having the greatest amount of relative movement arewhite and those having the least amount of relative movement are blackwith pixels having some level of movement between the maximum (white)and minimum (black) being a shade of gray.

Irrespective of the particular optical flow algorithm(s) employed, inblock 1220, the in-vehicle computing system 102 generates an image ofthe overlapping image region based on the identified moving objects. Insome embodiments, a combined image of the neighboring images may begenerated by appending the neighboring images to one another and, forexample, superimposing the generated image of the overlapping imageregion on the overlapping image region of the combined image. It shouldbe appreciated that if no moving objects are identified, the in-vehiclecomputing system 102 may utilize another technique for combining theneighboring images (e.g., image stitching) in some embodiments. However,if moving objects have been identified, the in-vehicle computing system102 attempts to preserve the moving objects in a resultant combinedimage of the neighboring images. Otherwise, projection of the combinedimage to the virtual bowl-shaped projection surface 402 may result ininvisible objects and/or blind spots.

In generating an image of the overlapping image region, the in-vehiclecomputing system 102 may generate an image that includes the identifiedmoving objects from one or both of the neighboring images. For example,in block 1222, the in-vehicle computing system 102 may generate an imagehaving unblended moving objects from one of the neighboring fisheyeimages. That is, if a moving object is identified in the overlappingimage region, the object is present in both of the neighboring images.As such, the in-vehicle computing system 102 may display the movingobject from only one of those images, however determined, anddeemphasize the moving object in the other neighboring image, forexample, by blurring the object (e.g., to avoid object duplicity). Inblock 1224, the in-vehicle computing system 102 may generate an imagehaving unblended moving objects from both of the neighboring fisheyeimages. For example, if the in-vehicle computing system 102 is unable todetermine which location of the moving object is more appropriate (i.e.,based on prospective projection locations of the object in each of theneighboring fisheye images onto the virtual bowl-shaped projectionsurface 402), the in-vehicle computing system 102 may determine todisplay the moving object in both locations by projecting the objectfrom both neighboring fisheye images onto the bowl-shaped projectionsurface 402. In block 1226, the in-vehicle computing system 102 maygenerate an image with the moving objects in an estimated position basedon the neighboring fisheye images. That is, the in-vehicle computingsystem 102 may determine a position at which to project each movingobject based on a suitable analysis (e.g., by averaging the projectionlocation of the object from the neighboring images). Additionally, insome embodiments, the in-vehicle computing system 102 may blend, filter,and/or otherwise modify the overlapping image region to reduce imageartifacts.

Referring back to FIG. 7, in block 712, the in-vehicle computing system102 performs object classification on one or more captured fisheyeimages. For example, the in-vehicle computing system 102 may determinethe locations of any pedestrians or animals in the surrounding vicinityof the vehicle 100. To do so, the in-vehicle computing system 102 mayexecute a method 1500 for classifying objects in a fisheye image asshown in FIG. 15. It should be appreciated that performing objectclassification on a fisheye image can be extremely difficult based onconventional classification algorithms due to the inherentcharacteristics of fisheye images. For example, fisheye images havestrong fisheye distortion, pixel resolution in a fisheye image decreasesnonlinearly from the center of the fisheye image to the borders, andlines in the real-world environment appear as curves in fisheye images.As such, objects appear with different shapes in fisheye image dependingon their relative position to the fisheye camera. Although fisheyedistortion may be compensated for, it typically results in perspectivedistortion, which also complicates conventional classification. Asdiscussed below, classification generally involves comparing an objectin an image to predefined object models or profiles for particularclasses of objects, which are typically defined in undistortedrectilinear space.

The illustrative method 1500 begins with block 1502 in which thein-vehicle computing system 102 determines whether to perform objectclassification. If so, the in-vehicle computing system 102 calibratesone or more of the fisheye cameras 126 in block 1504. As indicatedabove, the cameras 126 are calibrated (e.g., both intrinsically andextrinsically) using a suitable technique to ensure accurate imagecapturing and extraction of metric data. For example, fisheye images maybe related to the projection,

${{P \cdot X} = {\lambda \begin{bmatrix}\left( {{Au} + t} \right) \\{f\left( {u,v} \right)}\end{bmatrix}}},$

where P is the projection matrix, X is generally a vector description ofthe homogeneous coordinates (e.g., x, y, and z), u is the projectioninto the fisheye image, λ is a scaling factor corresponding with thehomogeneous ambiguity, A and t are factors corresponding withmisalignment (e.g., camera construction, resolution, pixel size, etc.),and f(u,v)=a₀+a₁p+a₂p²+ . . . +a_(n)p^(n) is a polynomial approximationof all types of distortions of the fisheye image, and where p is themetric distance between the point (u,v) and the optical axis. In someembodiments, calibration of a fisheye camera includes the determinationof the values of A, t, and f(u,v) in the projection equation providedabove. Of course, other calibration techniques may be used in otherembodiments to calibrate the cameras 126.

In block 1506, the in-vehicle computing system 102 determines a regionof interest in the fisheye image for which to perform objectclassification (see, for example, the fisheye image 1600 of FIG. 16). Asindicated above, performing object classification on the entire image iscomputationally intensive and projection of the entire fisheye image toa rectilinear image causes significant perspective distortion.Performing object classification on a region of the image (i.e., lessthan the entire image) reduces the effect of both of those factors. Itshould be appreciated that the location, size, and/or othercharacteristics of the region of interest may be determined using anysuitable techniques. For example, the size of the region of interest maybe selected in some embodiments based on the size of an object for whichclassification is to be performed (e.g., pedestrians).

In block 1508, the in-vehicle computing system 102 may determine aregion of interest by utilizing an optical flow algorithm to identifymoving objects as described above with regard to method 1200 of FIGS. 12and 13. That is, the in-vehicle computing system 102 may determine toperform object classification on regions of the fisheye image for whichthere is significant movement relative to the vehicle 100. In block1510, the in-vehicle computing system 102 may perform object detectionon the fisheye image to identify a region of interest. Althoughclassification algorithms typically perform very poorly on fisheyeimages, the in-vehicle computing system 102 may be able to determine,using various image filters and/or feature detectors on the fisheyeimage, for example, the existence of an object in the fisheye image(i.e., even if the in-vehicle computing system 102 is unable to classifythe detected object). In block 1512, the in-vehicle computing system 102may determine the region of interest by performing object detection on adistorted rectilinear image. In doing so, the in-vehicle computingsystem 102 may project the fisheye image onto a virtual hemisphericaldisplay surface in block 1514 and then to a plane therefrom. Asdiscussed above, a rectilinear image resulting from a projection of theentire image includes perspective distortion (see, for example, thedistorted rectilinear image 1700 of FIG. 17). However, object detectionalgorithms may, nonetheless, be employed on the distorted rectilinearimage to identify regions of interest for further analysis andclassification by the in-vehicle computing system 102.

In block 1516, the in-vehicle computing system 102 projects thedetermined region of interest onto a virtual hemispherical displaysurface to generate a hemispherical image region. Further, in block1518, the in-vehicle computing system 102 projects the hemisphericalimage region onto a plane perpendicular to a point in the hemisphericalimage region (e.g., the center point) to generate a rectilinear image.As described above, projection of the image region to the plane resultsin significantly less distortion (e.g., perspective distortion) thanprojection of the entire fisheye image onto the plane. As such, objectclassification algorithms will generally perform better (e.g., quickerand/or more accurately) with the rectilinear image (see, for example,the rectilinear image 1800 of FIG. 18) generated based on the region ofinterest than a rectilinear image (see FIG. 17) generated based on theentire image. In other embodiments, the in-vehicle computing system 102may employ a different combination of projections to project the regionof interest onto a plane to create a rectilinear image with minimalperspective distortion. For example, the in-vehicle computing system 102may project the region of interest directly from the fisheye image tothe plane (i.e., bypassing projection to/from the hemispherical surface)in some embodiments.

In block 1520, the in-vehicle computing system 102 performs objectclassification on the generated rectilinear image. As discussed above,in doing so, the in-vehicle computing system 102 compares therectilinear image or portions thereof (e.g., interest points) to one ormore predefined object profiles/models to determine whether there is amatch. Depending on the particular embodiments, there may be objectprofiles for a wide array of different types/classes of objects. Forexample, some object profiles may include adult pedestrians, childpedestrians, pets, vehicles, and others. It should be appreciated thatthe object profiles may be embodied as any suitable data structure andmay include attributes such as, for example, height, width, orientation,average speed, importance, identifiable interest points, and othersuitable characteristics. Regardless of whether the in-vehicle computingsystem 102 is able to successfully classify the object, the method 1500returns to block 1502 in which the in-vehicle computing system 102determines whether to perform object classification (e.g., on anotherfisheye image).

Referring back to FIG. 7, the in-vehicle computing system 102 displays abowl-shaped image in block 714. As discussed above, to do so, thein-vehicle computing system 102 generates, or otherwise uses, a virtualbowl-shaped projection surface 402 on which to project one or moreimages (e.g., fisheye images, combined/composite images, panoramicimages, etc.), thereby creating a “bowl-shaped” image. The projectionimproves, for example, driver visibility of the environment surroundingthe vehicle 100. Of course, as discussed above, in some embodiments, theone or more images may be modified to have a bowl shape consistent withthe virtual bowl-shaped projection surface 402 rather than be“projected” onto the bowl-shaped projection surface 402 in thetraditional sense. It should be appreciated that, in block 716, thein-vehicle computing system 102 may generate the bowl-shaped image basedon the combined images and an orientation of the virtual camera 202 asdiscussed above. For example, the images captured by the cameras 126 maybe combined to generate a single combined image that may be “projected”onto the virtual bowl-shaped projection surface 402 to generate thebowl-shaped image. As discussed above, in some embodiments, theprojection may be performed in conjunction with, for example, thestitching of neighboring images as described above with regard to themethod 900 of FIG. 9. Further, in some embodiments, the in-vehiclecomputing system 102 may only generate or display a portion of thebowl-shaped image and/or bowl-shaped projection surface 402 within thefield of view of, and from the same perspective as, the virtual camera202. Additionally, in block 718, the in-vehicle computing system 102 mayhighlight any objects that have been detected and classified by thein-vehicle computing system 102 as discussed above. Depending on theparticular embodiment, the in-vehicle computing system 102 may, forexample, visually highlight the classified object on the display 118 forthe user's convenience, transmit an audio alert to the user identifyingthe object's presence, and/or utilize one or more otherelectromechanical features of the vehicle 100 based on theclassification (e.g., automatically braking the vehicle 100).

In some embodiments, the in-vehicle computing system 102 may execute amethod 1900 as shown in FIG. 19 to generate a bowl-shaped image as shownin FIG. 4. The illustrative method 1900 begins with block 1902 in whichthe in-vehicle computing system 102 determines whether to generate abowl-shaped image. If so, the in-vehicle computing system 102 generates,or otherwise determines, the virtual bowl-shaped projection surface 402on which to project one or more images (e.g., combined images) andprojects the image(s) onto the bowl-shaped projection surface 402. Inparticular, in block 1904, the in-vehicle computing system 102determines the major and minor semi-axes of the vertical slice 408 (seeFIG. 4) of the virtual bowl-shaped projection surface 402 (i.e., to begenerated) along the length of the vehicle 100 (e.g., the vertical slicetaken across the bowl-shaped projection surface 402 along thelongitudinal axis of the vehicle 100). It should be appreciated that thein-vehicle computing system 102 may determine the appropriate size ofthe major and minor axes of the vertical slice 408 using any suitabletechniques (e.g., based on the size of the vehicle 100 or predeterminedvalues/thresholds).

In block 1906, the in-vehicle computing system 102 determines a“horizontal offset” of the bowl-shaped projection surface 402 to begenerated. In some embodiments, the horizontal offset may be apredetermined value stored on the memory 114 or the data storage 116 ofthe in-vehicle computing system 102 for generation of the virtualbowl-shaped projection surface 402. In the illustrative embodiment, thehorizontal offset has a length that is less that the length of thevehicle 100 and greater than half the length of the vehicle 100 (e.g.,slightly greater than half the length of the vehicle 100).

As discussed above, the in-vehicle computing system 102 may generateeach horizontal slice of the virtual bowl-shaped projection surface 402and combine the horizontal slices to generate the virtual bowl-shapedprojection surface 402. Accordingly, in block 1908, the in-vehiclecomputing system 102 selects the next horizontal slice of the virtualbowl-shaped projection surface 402 for generation. It should beappreciated that the “thickness” or width of the horizontal slices mayvary depending on the particular embodiment. For example, in someembodiments, the width of the horizontal slices may be infinitesimallysmall (i.e., virtually) or at a maximum resolution permitted byhardware, firmware, and/or software limitations of the in-vehiclecomputing system 102. Additionally, the order of selection of thehorizontal slices may also depend on the particular embodiment (e.g.,top to bottom, bottom to top, random selection, etc.).

In block 1910, the in-vehicle computing system 102 determines a firstsemi-axis (e.g., the length of the major semi-axis) of the selectedhorizontal slice. In doing so, in block 1912, the in-vehicle computingsystem 102 may determine, in some embodiments, the first semi-axisaccording to

${{aHorizontal} = {{aOffset} + {{aVertical}*\sqrt{1 - \left( \frac{z - {bVertical}}{bVertical} \right)^{2}}}}},$

where aHorizontal is the length of the first semi-axis (i.e., the majorsemi-axis) of the selected horizontal slice, aOffset is the horizontaloffset determined above, aVertical is the length of the major semi-axisof the vertical slice, bVertical is the length of the minor semi-axis ofthe vertical slice, and z is the height of the selected horizontal sliceabove the ground plane 404 of the bowl-shaped projection surface 402.

In block 1914, the in-vehicle computing system 102 determines anadaptive ratio for the selected horizontal slice. As discussed above, inthe illustrative embodiment, the adaptive ratio is predefined andchanges linearly across the horizontal slices such that the adaptiveratio is approximately unity (i.e., 1.0) at the top 412 (see FIG. 4) ofthe bowl-shaped projection surface 402 and approximately equal to avalue determined as the length of the car divided by the width of thecar

$\left( {{i.e.},\frac{carLength}{carWidth}} \right)$

at the bottom 410 (see FIG. 4) of the bowl-shaped projection surface402. Accordingly, the top horizontal slice of the virtual bowl-shapedprojection surface 402 is shaped as a circle and the bottom horizontalslice of the virtual bowl-shaped projection surface 402 (i.e., thehorizontal slice abutting the ground plane of the virtual bowl-shapedprojection surface 402) is shaped as a non-circular ellipse. In someembodiments, the adaptive ratio may be expressed as a linearone-dimensional function with an identifier of a horizontal slice beingthe input to the function and the functional output being used incalculations described herein. As such, each horizontal slice isassociated with a different adaptive ratio value.

In block 1916, the in-vehicle computing system 102 determines a secondsemi-axis (e.g., the length of the minor semi-axis) of the selectedhorizontal slice. In doing so, in block 1918, the in-vehicle computingsystem 102 may determine the second semi-axis according to

${{bHortizontal} = \frac{aHorizontal}{adaptiveRatio}},$

where bHorizontal is the length of the second semi-axis (i.e., the minorsemi-axis) of the selected horizontal slice, aHorizontal is the lengthof the first semi-axis (i.e., the major semi-axis) of the selectedhorizontal slice, and adaptiveRatio is the adaptive ratio for theselected horizontal slice as discussed above.

In block 1920, the in-vehicle computing system 102 determines whetherany horizontal slices of the virtual bowl-shaped projection surface 402remain for generation. If so, the method 1900 returns to block 1908 inwhich the in-vehicle computing system 102 selects the next horizontalslice for generation. However, if all of the horizontal slices of thevirtual bowl-shaped projection surface 402 have been generated, thein-vehicle computing system 102 combines the horizontal slices togenerate the bowl-shaped projection surface 402 in block 1922 using asuitable technique or algorithm. For example, the in-vehicle computingsystem 102 may “stack” the horizontal slices on one another in theappropriate order and merge the slices together to generate a singlevirtual surface. Additionally, in some embodiments, the in-vehiclecomputing system 102 also merges a flat ground plane 404 (i.e., thebottom of the bowl-shaped projection surface 402) to the bottom of thevirtual surface to generate the virtual bowl-shaped projection surface402.

In block 1924, the in-vehicle computing system 102 filters a bottomsection of the generated virtual-bowl shaped surface 402 adjacent theground plane 404 to smooth the junction between the ground plane 404 andthe lower horizontal slices of the bowl-shaped projection surface 402(see FIG. 6). As discussed above, the in-vehicle computing system 102may use any suitable image filters, algorithms, techniques, and/ormechanisms to do so. In block 1926, the in-vehicle computing system 102projects the combined image(s) onto the generated virtual bowl-shapedprojection surface 402 to generate the bowl-shaped image. As discussedabove, the bowl-shaped image improves the user's perception of theenvironment surrounding the vehicle 100. Additionally, it should beappreciated that images captured of the real-world ground plane may, insome embodiments, be projected directly back to the ground plane 404such that there minimal or no distortion. Further, as discussed above,the generated bowl-shaped image is displayed on the display 118 of thein-vehicle computing system 102 for the user of the in-vehicle computingsystem 102 (e.g., the driver of the vehicle 100).

Returning to FIG. 7, in block 720, the in-vehicle computing system 102determines whether any user input has been received (e.g., from thedriver) to modify the field of view of the virtual camera 202 (e.g., bytranslating or rotating the virtual camera 202). If so, the in-vehiclecomputing system 102 modifies the display of the bowl-shaped image basedon the user input and the new position of the virtual camera 202 inblock 722. For example, the user may rotate the virtual camera 202within the bowl-shaped image (e.g., from a perspective consistent withthat inside the vehicle 100) to display a different part of thebowl-shaped image. It should be appreciated that, in some embodiments,the in-vehicle computing system 102 may utilize multiple virtual cameras202 (e.g., to display multiple perspectives of the bowl-shaped imagesimultaneously). For example, in one embodiment, the in-vehiclecomputing system 102 may display a view of the bowl-shaped image fromeach of the four sides of the vehicle 100. In other embodiments, theentire bowl-shaped image or a portion thereof may be displayed on a 3Ddisplay as discussed above. In response to modifying the display of thebowl-shaped image (in block 722) or determining that no user input hasbeen received to modify the bowl-shaped image (in block 720), the method700 returns to block 702 in which the in-vehicle computing system 102determines whether to display the surrounding environment of the vehicle100.

Examples

Illustrative examples of the technologies disclosed herein are providedbelow. An embodiment of the technologies may include any one or more,and any combination of, the examples described below.

Example 1 includes a computing system for utilizing a bowl-shaped image,the computing system comprising a first fisheye camera to capture afirst fisheye image including a first scene; a second fisheye camera tocapture a second fisheye image including a second scene overlapping thefirst scene at an overlapping region; a combined image generation moduleto generate a combined image of the first fisheye image and the secondfisheye image; an object classification module to perform objectclassification on a region of interest of the at least one of the firstfisheye image or the second fisheye image to classify an object withinthe region of interest; and a bowl-generation module to generate aportion of a bowl-shaped image based on the combined image.

Example 2 includes the subject matter of Example 1, and wherein thecombined image generation module comprises a feature detection module toperform feature detection on the first fisheye image and the secondfisheye to detect one or more features located in the overlappingregion; a feature matching module to perform feature matching on thedetected features to determine one or more matched features of the firstfisheye image and the second fisheye image; a seam determination moduleto (i) determine a first seam of the first fisheye image in theoverlapping region based on the matched features of the first fisheyeimage and the second fisheye image, (ii) determine a second seam of thesecond fisheye image in the overlapping region based on the matchedfeatures of the first fisheye image and the second fisheye image, (iii)project the first fisheye image and the second fisheye image onto avirtual bowl-shaped projection surface to generate correspondingprojected images, and (iv) determine a third seam in an overlappingregion of the corresponding projected images based on correspondingprojections of the first seam and the second seam; and an imagesynthesis module to generate a combined image of the first fisheye imageand the second fisheye image by a combination of the projections of thefirst and second fisheye images at the determined third seam.

Example 3 includes the subject matter of any of Examples 1 and 2, andwherein the feature matching module is further to determine afundamental matrix of a camera system including the first fisheye camerathe second fisheye camera, wherein the fundamental matrix is based onthe geometry of the camera system.

Example 4 includes the subject matter of any of Examples 1-3, andwherein to perform feature matching comprises to filter feature matchcandidates based on the fundamental matrix.

Example 5 includes the subject matter of any of Examples 1-4, andwherein to perform feature matching comprises to minimize a Samson'sdistance of feature match candidates.

Example 6 includes the subject matter of any of Examples 1-5, andwherein to minimize the Samson's distance of the feature matchcandidates comprises to minimize an expression

${\sum\limits_{i}\frac{\left( {x_{i}^{\prime \; T}{Fx}_{i}} \right)^{2}}{\left( {Fx}_{i} \right)_{1}^{2} + \left( {Fx}_{i} \right)_{2}^{2} + \left( {Fx}_{i}^{\prime} \right)_{1}^{2} + \left( {Fx}_{i}^{\prime} \right)_{2}^{2}}},$

wherein x₁ and x₂ are coordinates of a pixel associated with a matchedfeature of the first fisheye image, x₁′ and x₂′ are coordinates of apixel associated with a matched feature of the second fisheye image, Fis the fundamental matrix, and i is an index of pixels associated withthe matched features.

Example 7 includes the subject matter of any of Examples 1-6, andwherein to determine the third seam comprises to determine a third seamas an average seam of the corresponding projections of the first seamand the second seam.

Example 8 includes the subject matter of any of Examples 1-7, andwherein the image synthesis module is further to perform at least one ofinterpolation or extrapolation of image pixels located between thecorresponding projections of the first and second seams in theoverlapping region.

Example 9 includes the subject matter of any of Examples 1-8, andwherein the image synthesis module is to deemphasize (i) an end of theprojection of the first image in the overlapping region and outside anarea defined between the corresponding projections of the first andsecond seams and (ii) an end of the projection of the second image inthe overlapping region and outside the area defined between thecorresponding projections of the first and second seam.

Example 10 includes the subject matter of any of Examples 1-9, andwherein the combined image generation module comprises an overlapdetermination module to determine an overlapping image region of thefirst fisheye image and the second fisheye image; a movementdetermination module to (i) determine movement of objects located in theoverlapping image region relative to the first and second fisheye camerabased on a plurality of fisheye images generated by the first and secondfisheye cameras and (ii) identify a moving object in the overlappingimage region based on the corresponding relative movement of theobjects; and an image generation module to generate an image of theoverlapping image region based on the identified moving object.

Example 11 includes the subject matter of any of Examples 1-10, andwherein to determine the overlapping image region comprises to performfeature detection on the first and second fisheye images to detect oneor more features located in the overlapping region; and perform featurematching on the detected features to determine one or more matchedfeatures of the first fisheye image and the second fisheye image.

Example 12 includes the subject matter of any of Examples 1-11, andwherein to determine the overlapping image region comprises to determinean overlapping image region based on a geometry of a camera systemincluding the first and second fisheye cameras.

Example 13 includes the subject matter of any of Examples 1-12, andwherein to determine the movement of objects comprises to apply agradient constraint equation to the plurality of fisheye images.

Example 14 includes the subject matter of any of Examples 1-13, andwherein to apply the gradient constraint equation comprises to determinea space derivative of intensity of pixels of the overlapping imageregion; determine a time derivative of intensity of pixels of theoverlapping image region for the plurality of fisheye images; and ensurethat ∇I(x,t)u+I_(t)(x,t)=0, wherein ∇I(x,t) is the space derivative, uis a two-dimensional velocity of the plurality of fisheye images, andI_(t)(x,t) is the time derivative.

Example 15 includes the subject matter of any of Examples 1-14, andwherein to generate the image of the overlapping image region comprisesto generate an image that includes the identified moving object from atleast one of the first fisheye image or the second fisheye image.

Example 16 includes the subject matter of any of Examples 1-15, andwherein to generate the image of the overlapping image region comprisesto generate an image that includes the identified moving object in anestimated location in the image based on corresponding locations of theidentified moving object in the first fisheye image and the secondfisheye image.

Example 17 includes the subject matter of any of Examples 1-16, andwherein the object classification module comprises a region of interestdetermination module to determine the region of interest in one of thefirst fisheye image or the second fisheye image; a rectilinearprojection module to (i) project the region of interest onto a virtualhemispherical display surface to generate a hemispherical image regionand (ii) project the hemispherical image region onto a planeperpendicular to a point in the hemispherical image region to generate arectilinear image; and an object identification module to perform objectclassification on the rectilinear image.

Example 18 includes the subject matter of any of Examples 1-17, andfurther including a calibration module to calibrate at least one of thefirst fisheye camera or the second fisheye camera to capture fisheyeimages.

Example 19 includes the subject matter of any of Examples 1-18, andwherein to determine the region of interest comprises to utilize anoptical flow algorithm to identify a moving object, the region ofinterest corresponding with an image region including the moving object.

Example 20 includes the subject matter of any of Examples 1-19, andwherein to determine the region of interest comprises to perform objectdetection on at least one of the first fisheye image or the secondfisheye image to identify an object of interest, the region of interestcorresponding with the identified object of interest.

Example 21 includes the subject matter of any of Examples 1-20, andwherein the region of interest determination module is further toproject the one of the first or second fisheye image onto a virtualhemispherical display surface to generate a hemispherical image, andproject the hemispherical image onto a plane to generate a perspectivelydistorted rectilinear image, wherein to determine the region of interestcomprises to perform object classification on the perspectivelydistorted rectilinear image.

Example 22 includes the subject matter of any of Examples 1-21, andwherein to perform object classification comprises to retrieve an objectprofile from a memory of the computing device; and compare an objectdetected in the rectilinear image to the retrieved object profile.

Example 23 includes the subject matter of any of Examples 1-22, andwherein the bowl-generation module comprises a bowl geometry module to(i) determine a major semi-axis and a minor semi-axis of a verticalslice of a virtual bowl-shaped projection surface to be generated, (ii)determine, for each horizontal slice of the virtual bowl-shapedprojection surface, a first semi-axis of the corresponding horizontalslice based on the major semi-axis and the minor semi-axis of thevertical slice, and (iii) determine, for each horizontal slice of thevirtual bowl-shaped projection surface, a second semi-axis of thecorresponding horizontal slice perpendicular to the first semi-axisbased on an adaptive ratio between the first semi-axis and the secondsemi-axis of the corresponding horizontal slice, wherein the adaptiveratio for the corresponding horizontal slice is different from eachadaptive ratio associated with the other horizontal slices; and a bowlsynthesis module to generate the virtual bowl-shaped projection surfacebased on a combination of the horizontal slices.

Example 24 includes the subject matter of any of Examples 1-23, andfurther including a bowl warping module to smooth a section of thegenerated virtual bowl-shaped projection surface adjacent a ground planeof the virtual bowl-shaped projection surface.

Example 25 includes the subject matter of any of Examples 1-24, andwherein the computing system comprises an in-vehicle computing system ofa vehicle; the bowl geometry module is further to determine a horizontaloffset of the corresponding horizontal slice, wherein the horizontaloffset is less than a length of the vehicle; and wherein to determinethe first semi-axis of the corresponding horizontal slice comprises todetermine, for each horizontal slice, the first semi-axis of thecorresponding horizontal slice based on the horizontal offset.

Example 26 includes the subject matter of any of Examples 1-25, andwherein the horizontal offset is greater than half the length of thevehicle.

Example 27 includes the subject matter of any of Examples 1-26, andwherein to determine the first semi-axis of the corresponding horizontalslice comprises to determine, for each horizontal slice, the firstsemi-axis of the corresponding horizontal slice as

${{aHorizontal} = {{aOffset} + {{aVertical}*\sqrt{1 - \left( \frac{z - {bVertical}}{bVertical} \right)^{2}}}}},$

wherein aHorizontal is a length of the first semi-axis, aOffset is thehorizontal offset, aVertical is a length of the major semi-axis of thevertical slice, bVertical is a length of the minor semi-axis of thevertical slice, and z is a height of the corresponding horizontal sliceabove a ground plane of the virtual bowl-shaped projection surface.

Example 28 includes the subject matter of any of Examples 1-27, andwherein the computing system comprises an in-vehicle computing system ofa vehicle; and the adaptive ratio changes linearly between adjacenthorizontal slices, the adaptive ratio is equal to one for a tophorizontal slice, and the adaptive ratio is a ratio of a length of thevehicle and a width of the vehicle for a bottom horizontal slice.

Example 29 includes the subject matter of any of Examples 1-28, andwherein a top horizontal slice of the virtual bowl-shaped projectionsurface has a circular geometric shape and a bottom horizontal slice ofthe virtual bowl-shaped projection surface has a non-circular ellipticalgeometric shape.

Example 30 includes the subject matter of any of Examples 1-29, andwherein to determine the second semi-axis of the correspondinghorizontal slice comprises to determine, for each horizontal slice, thesecond semi-axis of the corresponding horizontal slice as a ratio of thefirst semi-axis of the corresponding horizontal slice and the adaptiveratio for the corresponding horizontal slice.

Example 31 includes the subject matter of any of Examples 1-30, andfurther including a display module to project an image onto thegenerated virtual bowl-shaped projection surface to generate the portionof the bowl-shaped image; and display the portion of the bowl-shapedimage on a display of the computing device.

Example 32 includes the subject matter of any of Examples 1-31, andwherein to display the portion of the bowl-shaped image comprises todisplay a segment of the bowl-shaped image corresponding with a field ofview of a virtual camera of a vehicle.

Example 33 includes the subject matter of any of Examples 1-32, andwherein the computing device is embodied as an in-vehicle computingsystem.

Example 34 includes a computing device for generating a bowl-shapedimage, the computing device comprising a bowl geometry module to (i)determine a major semi-axis and a minor semi-axis of a vertical slice ofa virtual bowl-shaped projection surface to be generated, (ii)determine, for each horizontal slice of the virtual bowl-shapedprojection surface, a first semi-axis of the corresponding horizontalslice based on the major semi-axis and the minor semi-axis of thevertical slice, and (iii) determine, for each horizontal slice of thevirtual bowl-shaped projection surface, a second semi-axis of thecorresponding horizontal slice perpendicular to the first semi-axisbased on an adaptive ratio between the first semi-axis and the secondsemi-axis of the corresponding horizontal slice, wherein the adaptiveratio for the corresponding horizontal slice is different from eachadaptive ratio associated with the other horizontal slices; and a bowlsynthesis module to generate the virtual bowl-shaped projection surfacebased on a combination of the horizontal slices.

Example 35 includes the subject matter of Example 34, and furtherincluding a bowl warping module to smooth a section of the generatedvirtual bowl-shaped projection surface adjacent a ground plane of thevirtual bowl-shaped projection surface.

Example 36 includes the subject matter of any of Examples 34 and 35, andwherein the computing device comprises an in-vehicle computing system ofa vehicle; the bowl geometry module is further to determine a horizontaloffset of the corresponding horizontal slice, wherein the horizontaloffset is less than a length of the vehicle; and wherein to determinethe first semi-axis of the corresponding horizontal slice comprises todetermine, for each horizontal slice, the first semi-axis of thecorresponding horizontal slice based on the horizontal offset.

Example 37 includes the subject matter of any of Examples 34-36, andwherein the horizontal offset is greater than half the length of thevehicle.

Example 38 includes the subject matter of any of Examples 34-37, andwherein to determine the first semi-axis of the corresponding horizontalslice comprises to determine, for each horizontal slice, the firstsemi-axis of the corresponding horizontal slice as

${{aHorizontal} = {{aOffset} + {{aVertical}*\sqrt{1 - \left( \frac{z - {bVertical}}{bVertical} \right)^{2}}}}},$

wherein aHorizontal is a length of the first semi-axis, aOffset is thehorizontal offset, aVertical is a length of the major semi-axis of thevertical slice, bVertical is a length of the minor semi-axis of thevertical slice, and z is a height of the corresponding horizontal sliceabove a ground plane of the virtual bowl-shaped projection surface.

Example 39 includes the subject matter of any of Examples 34-38, andwherein the computing device comprises an in-vehicle computing system ofa vehicle; and the adaptive ratio changes linearly between adjacenthorizontal slices, the adaptive ratio is equal to one for a tophorizontal slice, and the adaptive ratio is a ratio of a length of thevehicle and a width of the vehicle for a bottom horizontal slice.

Example 40 includes the subject matter of any of Examples 34-39, andwherein a top horizontal slice of the virtual bowl-shaped projectionsurface has a circular geometric shape and a bottom horizontal slice ofthe virtual bowl-shaped projection surface has a non-circular ellipticalgeometric shape.

Example 41 includes the subject matter of any of Examples 34-40, andwherein to determine the second semi-axis of the correspondinghorizontal slice comprises to determine, for each horizontal slice, thesecond semi-axis of the corresponding horizontal slice as a ratio of thefirst semi-axis of the corresponding horizontal slice and the adaptiveratio for the corresponding horizontal slice.

Example 42 includes the subject matter of any of Examples 34-41, andfurther including a display module to project an image onto thegenerated virtual bowl-shaped projection surface to generate thebowl-shaped image; and display the bowl-shaped image on a display of thecomputing device.

Example 43 includes the subject matter of any of Examples 34-42, andwherein to display the bowl-shaped image comprises to display a segmentof the bowl-shaped image corresponding with a field of view of a virtualcamera of a vehicle.

Example 44 includes the subject matter of any of Examples 34-43, andwherein the computing device is embodied as an in-vehicle computingsystem.

Example 45 includes a computing device for combining fisheye imagesgenerated by a plurality of fisheye cameras, the computing devicecomprising a feature detection module to perform feature detection on afirst fisheye image capturing a first scene and on a second fisheyeimage capturing a second scene overlapping with the first scene at anoverlapping region of the first and second scenes to detect one or morefeatures located in the overlapping region; a feature matching module toperform feature matching on the detected features to determine one ormore matched features of the first fisheye image and the second fisheyeimage; a seam determination module to (i) determine a first seam of thefirst fisheye image in the overlapping region based on the matchedfeatures of the first fisheye image and the second fisheye image, (ii)determine a second seam of the second fisheye image in the overlappingregion based on the matched features of the first fisheye image and thesecond fisheye image, (iii) project the first fisheye image and thesecond fisheye image onto a virtual bowl-shaped projection surface togenerate corresponding projected images, and (iv) determine a third seamin an overlapping region of the corresponding projected images based oncorresponding projections of the first seam and the second seam; and animage synthesis module to generate a combined image of the first fisheyeimage and the second fisheye image by a combination of the projectionsof the first and second fisheye images at the determined third seam.

Example 46 includes the subject matter of Example 45, and wherein thefeature matching module is further to determine a fundamental matrix ofa camera system including a first fisheye camera to generate the firstfisheye image and a second fisheye camera to generate the second fisheyeimage, wherein the fundamental matrix is based on the geometry of thecamera system.

Example 47 includes the subject matter of any of Examples 45 and 46, andwherein to perform feature matching comprises to filter feature matchcandidates based on the fundamental matrix.

Example 48 includes the subject matter of any of Examples 45-47, andwherein to perform feature matching comprises to minimize a Samson'sdistance of feature match candidates.

Example 49 includes the subject matter of any of Examples 45-48, andwherein to minimize the Samson's distance of the feature matchcandidates comprises to minimize an expression

${\sum\limits_{i}\frac{\left( {x_{i}^{\prime \; T}{Fx}_{i}} \right)^{2}}{\left( {Fx}_{i} \right)_{1}^{2} + \left( {Fx}_{i} \right)_{2}^{2} + \left( {Fx}_{i}^{\prime} \right)_{1}^{2} + \left( {Fx}_{i}^{\prime} \right)_{2}^{2}}},$

wherein x₁ and x₂ are coordinates of a pixel associated with a matchedfeature of the first fisheye image, x₁′ and x₂′ are coordinates of apixel associated with a matched feature of the second fisheye image, Fis the fundamental matrix, and i is an index of pixels associated withthe matched features.

Example 50 includes the subject matter of any of Examples 45-49, andwherein to determine the third seam comprises to determine a third seamas an average seam of the corresponding projections of the first seamand the second seam.

Example 51 includes the subject matter of any of Examples 45-50, andwherein the image synthesis module is further to perform at least one ofinterpolation or extrapolation of image pixels located between thecorresponding projections of the first and second seams in theoverlapping region.

Example 52 includes the subject matter of any of Examples 45-51, andwherein the image synthesis module is to deemphasize (i) an end of theprojection of the first image in the overlapping region and outside anarea defined between the corresponding projections of the first andsecond seams and (ii) an end of the projection of the second image inthe overlapping region and outside the area defined between thecorresponding projections of the first and second seam.

Example 53 includes a computing device for combining fisheye imagesbased on optical flow, the computing device comprising an overlapdetermination module to determine an overlapping image region of a firstfisheye image generated by a first fisheye camera and a second fisheyeimage generated by a second fisheye camera; a movement determinationmodule to (i) determine movement of objects located in the overlappingimage region relative to the first and second fisheye camera based on aplurality of fisheye images generated by the first and second fisheyecameras and (ii) identify a moving object in the overlapping imageregion based on the corresponding relative movement of the objects; andan image generation module to generate an image of the overlapping imageregion based on the identified moving object.

Example 54 includes the subject matter of Example 53, and wherein todetermine the overlapping image region comprises to perform featuredetection on the first and second fisheye image, wherein the firstfisheye image captures a first scene and the second fisheye imagecaptures a second scene overlapping with the first scene at anoverlapping region of the first and second scenes to detect one or morefeatures located in the overlapping region; and perform feature matchingon the detected features to determine one or more matched features ofthe first fisheye image and the second fisheye image.

Example 55 includes the subject matter of any of Examples 53 and 54, andwherein to determine the overlapping image region comprises to determinean overlapping image region based on a geometry of a camera systemincluding the first and second fisheye cameras.

Example 56 includes the subject matter of any of Examples 53-55, andwherein to determine the movement of objects comprises to apply agradient constraint equation to the plurality of fisheye images.

Example 57 includes the subject matter of any of Examples 53-56, andwherein to apply the gradient constraint equation comprises to determinea space derivative of intensity of pixels of the overlapping imageregion; determine a time derivative of intensity of pixels of theoverlapping image region for the plurality of fisheye images; and ensurethat ∇I(x,t)u+I_(t)(x,t)=0, wherein ∇I(x,t) is the space derivative, uis a two-dimensional velocity of the plurality of fisheye images, andI_(t)(x,t) is the time derivative.

Example 58 includes the subject matter of any of Examples 53-57, andwherein to generate the image of the overlapping image region comprisesto generate an image that includes the identified moving object from atleast one of the first fisheye image or the second fisheye image.

Example 59 includes the subject matter of any of Examples 53-58, andwherein to generate the image of the overlapping image region comprisesto generate an image that includes the identified moving object in anestimated location in the image based on corresponding locations of theidentified moving object in the first fisheye image and the secondfisheye image.

Example 60 includes a computing device for classifying objects in afisheye image, the computing device comprising a region of interestdetermination module to determine a region of interest in a fisheyeimage; a rectilinear projection module to (i) project the region ofinterest onto a virtual hemispherical display surface to generate ahemispherical image region and (ii) project the hemispherical imageregion onto a plane perpendicular to a point in the hemispherical imageregion to generate a rectilinear image; and an object identificationmodule to perform object classification on the rectilinear image.

Example 61 includes the subject matter of Example 60, and furtherincluding a calibration module to calibrate a fisheye camera to capturethe fisheye image.

Example 62 includes the subject matter of any of Examples 60 and 61, andwherein to determine the region of interest comprises to utilize anoptical flow algorithm to identify a moving object, the region ofinterest corresponding with an image region including the moving object.

Example 63 includes the subject matter of any of Examples 60-62, andwherein to determine the region of interest comprises to perform objectdetection on the fisheye image to identify an object of interest, theregion of interest corresponding with the identified object of interest.

Example 64 includes the subject matter of any of Examples 60-63, andwherein the region of interest determination module is further toproject the fisheye image onto a virtual hemispherical display surfaceto generate a hemispherical image, and project the hemispherical imageonto a plane to generate a perspectively distorted rectilinear image,wherein to determine the region of interest comprises to perform objectclassification on the perspectively distorted rectilinear image.

Example 65 includes the subject matter of any of Examples 60-64, andwherein to perform object classification comprises to retrieve an objectprofile from a memory of the computing device; and compare an objectdetected in the rectilinear image to the retrieved object profile.

Example 66 includes a method for utilizing a bowl-shaped image on acomputing device, the method comprising receiving, by the computingdevice, a first fisheye image capturing a first scene and a secondfisheye image capturing a second scene overlapping with the first sceneat an overlapping region; generating, by the computing device, acombined image of the first fisheye image and the second fisheye image;performing, by the computing device, object classification on a regionof interest of at least one of the first fisheye image or the secondfisheye image to classify an object within the region of interest; andgenerating, by the computing device, a portion of a bowl-shaped imagebased on the combined image.

Example 67 includes the subject matter of Example 66, and whereingenerating the combined image comprises performing feature detection onthe first fisheye image capturing and the second fisheye image to detectone or more features located in the overlapping region; performingfeature matching on the detected features to determine one or morematched features of the first fisheye image and the second fisheyeimage; determining a first seam of the first fisheye image in theoverlapping region based on the matched features of the first fisheyeimage and the second fisheye image; determining a second seam of thesecond fisheye image in the overlapping region based on the matchedfeatures of the first fisheye image and the second fisheye image;determining a third seam in the overlapping region based on the firstseam and the second seam; and generating a combined image of the firstfisheye image and the second fisheye image by combining the first andsecond fisheye images at the determined third seam.

Example 68 includes the subject matter of any of Examples 66 and 67, andwherein generating the combined image comprises determining afundamental matrix of a camera system including a first fisheye camerato generate the first fisheye image and a second fisheye camera togenerate the second fisheye image, wherein the fundamental matrix isbased on the geometry of the camera system.

Example 69 includes the subject matter of any of Examples 66-68, andwherein performing feature matching comprises filtering feature matchcandidates based on the fundamental matrix.

Example 70 includes the subject matter of any of Examples 66-69, andwherein performing feature matching comprises minimizing a Samson'sdistance of feature match candidates.

Example 71 includes the subject matter of any of Examples 66-70, andwherein minimizing the Samson's distance of the feature match candidatescomprises minimizing an expression

${\sum\limits_{i}\frac{\left( {x_{i}^{\prime \; T}{Fx}_{i}} \right)^{2}}{\left( {Fx}_{i} \right)_{1}^{2} + \left( {Fx}_{i} \right)_{2}^{2} + \left( {Fx}_{i}^{\prime} \right)_{1}^{2} + \left( {Fx}_{i}^{\prime} \right)_{2}^{2}}},$

wherein x₁ and x₂ are coordinates of a pixel associated with a matchedfeature of the first fisheye image, x₁′ and x₂′ are coordinates of apixel associated with a matched feature of the second fisheye image, Fis the fundamental matrix, and i is an index of pixels associated withthe matched features.

Example 72 includes the subject matter of any of Examples 66-71, andwherein determining the third seam comprises determining a third seam asan average seam of the corresponding projections of the first seam andthe second seam.

Example 73 includes the subject matter of any of Examples 66-72, andwherein generating the combined image comprises performing at least oneof interpolation or extrapolation of image pixels located between thecorresponding projections of the first and second seams in theoverlapping region.

Example 74 includes the subject matter of any of Examples 66-73, andwherein generating the combined image comprises deemphasizing (i) an endof the projection of the first image in the overlapping region andoutside an area defined between the corresponding projections of thefirst and second seams and (ii) an end of the projection of the secondimage in the overlapping region and outside the area defined between thecorresponding projections of the first and second seam.

Example 75 includes the subject matter of any of Examples 66-74, andwherein generating the combined image comprises determining anoverlapping image region of the first fisheye image and the secondfisheye images, the first fisheye image being generated by a firstfisheye camera and the second fisheye image being generated by a secondfisheye camera; determining movement of objects located in theoverlapping image region relative to the first and second fisheye camerabased on a plurality of fisheye images generated by the first and secondfisheye cameras; identifying a moving object in the overlapping imageregion based on the corresponding relative movement of the objects; andgenerating an image of the overlapping image region based on theidentified moving object.

Example 76 includes the subject matter of any of Examples 66-75, andwherein determining the overlapping image region comprises performingfeature detection on the first and second fisheye image to detect one ormore features located in the overlapping region; and performing featurematching on the detected features to determine one or more matchedfeatures of the first fisheye image and the second fisheye image.

Example 77 includes the subject matter of any of Examples 66-76, andwherein determining the overlapping image region comprises determiningan overlapping image region based on a geometry of a camera systemincluding the first and second fisheye cameras.

Example 78 includes the subject matter of any of Examples 66-77, andwherein determining the movement of objects comprises applying agradient constraint equation to the plurality of fisheye images.

Example 79 includes the subject matter of any of Examples 66-78, andwherein determining the movement of objects comprises determining aspace derivative of intensity of pixels of the overlapping image region;determining a time derivative of intensity of pixels of the overlappingimage region for the plurality of fisheye images; and ensuring that∇I(x,t)u+I_(t)(x,t)=0, wherein ∇I(x,t) is the space derivative, u is atwo-dimensional velocity of the plurality of fisheye images, andI_(t)(x,t) is the time derivative.

Example 80 includes the subject matter of any of Examples 66-79, andwherein generating the image of the overlapping image region comprisesgenerating an image including the identified moving object from at leastone of the first fisheye image or the second fisheye image.

Example 81 includes the subject matter of any of Examples 66-80, andwherein generating the image of the overlapping image region comprisesgenerating an image including the identified moving object in anestimated location in the image based on corresponding locations of theidentified moving object in the first fisheye image and the secondfisheye image.

Example 82 includes the subject matter of any of Examples 66-81, andwherein performing object classification comprises determining theregion of interest in one of the first fisheye image or the secondfisheye image; generating a rectilinear image by (i) projecting theregion of interest onto a virtual hemispherical display surface togenerate a hemispherical image region and (ii) projecting thehemispherical image region onto a plane perpendicular to a point in thehemispherical image region; and classifying an object in the rectilinearimage.

Example 83 includes the subject matter of any of Examples 66-82, andwherein performing object classification comprises calibrating a fisheyecamera to capture one of the first fisheye image or the second fisheyeimage.

Example 84 includes the subject matter of any of Examples 66-83, andwherein determining the region of interest comprises utilizing anoptical flow algorithm to identify a moving object, the region ofinterest corresponding with an image region including the moving object.

Example 85 includes the subject matter of any of Examples 66-84, andwherein determining the region of interest comprises performing objectdetection on at least one of the first fisheye image or the secondfisheye image to identify an object of interest, the region of interestcorresponding with the identified object of interest.

Example 86 includes the subject matter of any of Examples 66-85, andwherein performing object classification comprises projecting the one ofthe first or second fisheye image onto a virtual hemispherical displaysurface to generate a hemispherical image, and projecting thehemispherical image onto a plane to generate a perspectively distortedrectilinear image, wherein determining the region of interest comprisesperforming object classification on the perspectively distortedrectilinear image.

Example 87 includes the subject matter of any of Examples 66-86, andwherein classifying the object comprises retrieving an object profilefrom a memory of the computing device; and comparing an object detectedin the rectilinear image to the retrieved object profile.

Example 88 includes the subject matter of any of Examples 66-87, andwherein generating the portion of the bowl-shaped image comprisesdetermining a major semi-axis and a minor semi-axis of a vertical sliceof a virtual bowl-shaped projection surface to be generated;determining, for each horizontal slice of the virtual bowl-shapedprojection surface, a first semi-axis of the corresponding horizontalslice based on the major semi-axis and the minor semi-axis of thevertical slice; determining, for each horizontal slice of the virtualbowl-shaped projection surface, a second semi-axis of the correspondinghorizontal slice perpendicular to the first semi-axis based on anadaptive ratio between the first semi-axis and the second semi-axis ofthe corresponding horizontal slice, the adaptive ratio for thecorresponding horizontal slice being different from each adaptive ratioassociated with the other horizontal slices; and generating the virtualbowl-shaped projection surface based on a combination of the horizontalslices.

Example 89 includes the subject matter of any of Examples 66-88, andwherein generating the portion of the bowl-shaped image comprisessmoothing a section of the generated virtual bowl-shaped projectionsurface adjacent a ground plane of the virtual bowl-shaped projectionsurface.

Example 90 includes the subject matter of any of Examples 66-89, andwherein the computing device comprises an in-vehicle computing system ofa vehicle; and generating the portion of the bowl-shaped image comprisesdetermining a horizontal offset of the corresponding horizontal slice,the horizontal offset being less than a length of the vehicle, whereindetermining the first semi-axis of the corresponding horizontal slicecomprises determining, for each horizontal slice, the first semi-axis ofthe corresponding horizontal slice based on the horizontal offset.

Example 91 includes the subject matter of any of Examples 66-90, andwherein the horizontal offset is greater than half the length of thevehicle.

Example 92 includes the subject matter of any of Examples 66-91, andwherein determining the first semi-axis of the corresponding horizontalslice comprises determining, for each horizontal slice, the firstsemi-axis of the corresponding horizontal slice as

${{aHorizontal} = {{aOffset} + {{aVertical}*\sqrt{1 - \left( \frac{z - {bVertical}}{bVertical} \right)^{2}}}}},$

wherein aHorizontal is a length of the first semi-axis, aOffset is thehorizontal offset, aVertical is a length of the major semi-axis of thevertical slice, bVertical is a length of the minor semi-axis of thevertical slice, and z is a height of the corresponding horizontal sliceabove a ground plane of the virtual bowl-shaped projection surface.

Example 93 includes the subject matter of any of Examples 66-92, andwherein the computing device comprises an in-vehicle computing system ofa vehicle; and the adaptive ratio changes linearly between adjacenthorizontal slices, the adaptive ratio is equal to one for a tophorizontal slice, and the adaptive ratio is a ratio of a length of thevehicle and a width of the vehicle for a bottom horizontal slice.

Example 94 includes the subject matter of any of Examples 66-93, andwherein a top horizontal slice of the virtual bowl-shaped projectionsurface has a circular geometric shape and a bottom horizontal slice ofthe virtual bowl-shaped projection surface has a non-circular ellipticalgeometric shape.

Example 95 includes the subject matter of any of Examples 66-94, andwherein determining the second semi-axis of the corresponding horizontalslice comprises determining, for each horizontal slice, the secondsemi-axis of the corresponding horizontal slice as a ratio of the firstsemi-axis of the corresponding horizontal slice and the adaptive ratiofor the corresponding horizontal slice.

Example 96 includes the subject matter of any of Examples 66-95, andwherein generating the portion of the bowl-shaped image comprisesprojecting an image onto the generated virtual bowl-shaped projectionsurface to generate the portion of the bowl-shaped image; and furthercomprising displaying, on a display of the computing device, the portionof the bowl-shaped image.

Example 97 includes the subject matter of any of Examples 66-96, andwherein displaying the portion of the bowl-shaped image comprisesdisplaying a segment of the bowl-shaped image corresponding with a fieldof view of a virtual camera of a vehicle.

Example 98 includes a computing device comprising a processor; and amemory having stored therein a plurality of instructions that whenexecuted by the processor cause the computing device to perform themethod of any of Examples 66-97.

Example 99 includes one or more machine-readable storage mediacomprising a plurality of instructions stored thereon that, in responseto being executed, result in a computing device performing the method ofany of Examples 66-97.

Example 100 includes a computing device for utilizing a bowl-shapedimage, the computing device comprising means for performing the methodof any of Examples 66-97.

Example 101 includes a method for generating a bowl-shaped image on acomputing device, the method comprising determining, by the computingdevice, a major semi-axis and a minor semi-axis of a vertical slice of avirtual bowl-shaped projection surface to be generated; determining, bythe computing device and for each horizontal slice of the virtualbowl-shaped projection surface, a first semi-axis of the correspondinghorizontal slice based on the major semi-axis and the minor semi-axis ofthe vertical slice; determining, by the computing device and for eachhorizontal slice of the virtual bowl-shaped projection surface, a secondsemi-axis of the corresponding horizontal slice perpendicular to thefirst semi-axis based on an adaptive ratio between the first semi-axisand the second semi-axis of the corresponding horizontal slice, theadaptive ratio for the corresponding horizontal slice being differentfrom each adaptive ratio associated with the other horizontal slices;and generating, by the computing device, the virtual bowl-shapedprojection surface based on a combination of the horizontal slices.

Example 102 includes the subject matter of Example 101, and furtherincluding smoothing, by the computing device, a section of the generatedvirtual bowl-shaped projection surface adjacent a ground plane of thevirtual bowl-shaped projection surface.

Example 103 includes the subject matter of any of Examples 101 and 102,and wherein the computing device comprises an in-vehicle computingsystem of a vehicle; and further comprising determining, by thecomputing device, a horizontal offset of the corresponding horizontalslice, the horizontal offset being less than a length of the vehicle,wherein determining the first semi-axis of the corresponding horizontalslice comprises determining, for each horizontal slice, the firstsemi-axis of the corresponding horizontal slice based on the horizontaloffset.

Example 104 includes the subject matter of any of Examples 101-103, andwherein the horizontal offset is greater than half the length of thevehicle.

Example 105 includes the subject matter of any of Examples 101-104, andwherein determining the first semi-axis of the corresponding horizontalslice comprises determining, for each horizontal slice, the firstsemi-axis of the corresponding horizontal slice as

${{aHorizontal} = {{aOffset} + {{aVertical}*\sqrt{1 - \left( \frac{z - {bVertical}}{bVertical} \right)^{2}}}}},$

wherein aHorizontal is a length of the first semi-axis, aOffset is thehorizontal offset, aVertical is a length of the major semi-axis of thevertical slice, bVertical is a length of the minor semi-axis of thevertical slice, and z is a height of the corresponding horizontal sliceabove a ground plane of the virtual bowl-shaped projection surface.

Example 106 includes the subject matter of any of Examples 101-105, andwherein the computing device comprises an in-vehicle computing system ofa vehicle; and the adaptive ratio changes linearly between adjacenthorizontal slices, the adaptive ratio is equal to one for a tophorizontal slice, and the adaptive ratio is a ratio of a length of thevehicle and a width of the vehicle for a bottom horizontal slice.

Example 107 includes the subject matter of any of Examples 101-106, andwherein a top horizontal slice of the virtual bowl-shaped projectionsurface has a circular geometric shape and a bottom horizontal slice ofthe virtual bowl-shaped projection surface has a non-circular ellipticalgeometric shape.

Example 108 includes the subject matter of any of Examples 101-107, andwherein determining the second semi-axis of the corresponding horizontalslice comprises determining, for each horizontal slice, the secondsemi-axis of the corresponding horizontal slice as a ratio of the firstsemi-axis of the corresponding horizontal slice and the adaptive ratiofor the corresponding horizontal slice.

Example 109 includes the subject matter of any of Examples 101-108, andfurther including projecting, by the computing device, an image onto thegenerated virtual bowl-shaped projection surface to generate thebowl-shaped image; and displaying, on a display of the computing device,the bowl-shaped image.

Example 110 includes the subject matter of any of Examples 101-109, andwherein displaying the bowl-shaped image comprises displaying a segmentof the bowl-shaped image corresponding with a field of view of a virtualcamera of a vehicle.

Example 111 includes a computing device comprising a processor; and amemory having stored therein a plurality of instructions that whenexecuted by the processor cause the computing device to perform themethod of any of Examples 101-110.

Example 112 includes one or more machine-readable storage mediacomprising a plurality of instructions stored thereon that, in responseto being executed, result in a computing device performing the method ofany of Examples 101-110.

Example 113 includes a computing device for generating a bowl-shapedimage, the computing device comprising means for performing the methodof any of Examples 101-10.

Example 114 includes a method for combining fisheye images, generated bya plurality of fisheye cameras, on a computing device, the methodcomprising performing, by the computing device, feature detection on afirst fisheye image capturing a first scene and on a second fisheyeimage capturing a second scene overlapping with the first scene at anoverlapping region of the first and second scenes to detect one or morefeatures located in the overlapping region; performing, by the computingdevice, feature matching on the detected features to determine one ormore matched features of the first fisheye image and the second fisheyeimage; determining, by the computing device, a first seam of the firstfisheye image in the overlapping region based on the matched features ofthe first fisheye image and the second fisheye image; determining, bythe computing device, a second seam of the second fisheye image in theoverlapping region based on the matched features of the first fisheyeimage and the second fisheye image; projecting, by the computing device,the first fisheye image and the second fisheye image onto a virtualbowl-shaped projection surface to generated corresponding projectedimages; determining, by the computing device, a third seam in anoverlapping region of the corresponding projected images based oncorresponding projections of the first seam and the second seam; andgenerating, by the computing device, a combined image of the firstfisheye image and the second fisheye image by combining the projectionsof the first and second fisheye images at the determined third seam.

Example 115 includes the subject matter of Example 114, and furtherincluding determining, by the computing device, a fundamental matrix ofa camera system including a first fisheye camera to generate the firstfisheye image and a second fisheye camera to generate the second fisheyeimage, wherein the fundamental matrix is based on the geometry of thecamera system.

Example 116 includes the subject matter of any of Examples 114 and 115,and wherein performing feature matching comprises filtering featurematch candidates based on the fundamental matrix.

Example 117 includes the subject matter of any of Examples 114-116, andwherein performing feature matching comprises minimizing a Samson'sdistance of feature match candidates.

Example 118 includes the subject matter of any of Examples 114-117, andwherein minimizing the Samson's distance of the feature match candidatescomprises minimizing an expression

${\sum\limits_{i}\frac{\left( {x_{i}^{\prime \; T}{Fx}_{i}} \right)^{2}}{\left( {Fx}_{i} \right)_{1}^{2} + \left( {Fx}_{i} \right)_{2}^{2} + \left( {Fx}_{i}^{\prime} \right)_{1}^{2} + \left( {Fx}_{i}^{\prime} \right)_{2}^{2}}},$

wherein x₁ and x₂ are coordinates of a pixel associated with a matchedfeature of the first fisheye image, x₁′ and x₂′ are coordinates of apixel associated with a matched feature of the second fisheye image, Fis the fundamental matrix, and i is an index of pixels associated withthe matched features.

Example 119 includes the subject matter of any of Examples 114-118, andwherein determining the third seam comprises determining a third seam asan average seam of the corresponding projections of the first seam andthe second seam.

Example 120 includes the subject matter of any of Examples 114-119, andfurther including performing, by the computing device, at least one ofinterpolation or extrapolation of image pixels located between thecorresponding projections of the first and second seams in theoverlapping region.

Example 121 includes the subject matter of any of Examples 114-120, andfurther including deemphasizing, by the computing device, (i) an end ofthe projection of the first image in the overlapping region and outsidean area defined between the corresponding projections of the first andsecond seams and (ii) an end of the projection of the second image inthe overlapping region and outside the area defined between thecorresponding projections of the first and second seam.

Example 122 includes a computing device comprising a processor; and amemory having stored therein a plurality of instructions that whenexecuted by the processor cause the computing device to perform themethod of any of Examples 114-121.

Example 123 includes one or more machine-readable storage mediacomprising a plurality of instructions stored thereon that, in responseto being executed, result in a computing device performing the method ofany of Examples 114-121.

Example 124 includes a computing device for combining fisheye imagesbased on image stitching, the computing device comprising means forperforming the method of any of Examples 114-121.

Example 125 includes a method for combining fisheye images on acomputing device based on optical flow, the method comprisingdetermining, by the computing device, an overlapping image region of afirst fisheye image generated by a first fisheye camera and a secondfisheye image generated by a second fisheye camera; determining, by thecomputing device, movement of objects located in the overlapping imageregion relative to the first and second fisheye camera based on aplurality of fisheye images generated by the first and second fisheyecameras; identifying, by the computing device, a moving object in theoverlapping image region based on the corresponding relative movement ofthe objects; and generating, by the computing device, an image of theoverlapping image region based on the identified moving object.

Example 126 includes the subject matter of Example 125, and whereindetermining the overlapping image region comprises performing, by thecomputing device, feature detection on the first and second fisheyeimage, the first fisheye image capturing a first scene and the secondfisheye image capturing a second scene overlapping with the first sceneat an overlapping region of the first and second scenes to detect one ormore features located in the overlapping region; and performing, by thecomputing device, feature matching on the detected features to determineone or more matched features of the first fisheye image and the secondfisheye image.

Example 127 includes the subject matter of any of Examples 125 and 126,and wherein determining the overlapping image region comprisesdetermining an overlapping image region based on a geometry of a camerasystem including the first and second fisheye cameras.

Example 128 includes the subject matter of any of Examples 125-127, andwherein determining the movement of objects comprises applying agradient constraint equation to the plurality of fisheye images.

Example 129 includes the subject matter of any of Examples 125-128, andwherein determining the movement of objects comprises determining aspace derivative of intensity of pixels of the overlapping image region;determining a time derivative of intensity of pixels of the overlappingimage region for the plurality of fisheye images; and ensuring that∇I(x,t)u+I_(t)(x,t)=0, wherein ∇I(x,t) is the space derivative, u is atwo-dimensional velocity of the plurality of fisheye images, andI_(t)(x,t) is the time derivative.

Example 130 includes the subject matter of any of Examples 125-129, andwherein generating the image of the overlapping image region comprisesgenerating an image including the identified moving object from at leastone of the first fisheye image or the second fisheye image.

Example 131 includes the subject matter of any of Examples 125-130, andwherein generating the image of the overlapping image region comprisesgenerating an image including the identified moving object in anestimated location in the image based on corresponding locations of theidentified moving object in the first fisheye image and the secondfisheye image.

Example 132 includes a computing device comprising a processor; and amemory having stored therein a plurality of instructions that whenexecuted by the processor cause the computing device to perform themethod of any of Examples 125-131.

Example 133 includes one or more machine-readable storage mediacomprising a plurality of instructions stored thereon that, in responseto being executed, result in a computing device performing the method ofany of Examples 125-131.

Example 134 includes a computing device for combining fisheye imagesbased on optical flow, the computing device comprising means forperforming the method of any of Examples 125-131.

Example 135 includes a method for classifying objects in a fisheye imageon a computing device, the method comprising determining, by thecomputing device, a region of interest in a fisheye image; generating,by the computing device, a rectilinear image by (i) projecting theregion of interest onto a virtual hemispherical display surface togenerate a hemispherical image region and (ii) projecting thehemispherical image region onto a plane perpendicular to a point in thehemispherical image region; and performing, by the computing device,object classification on the rectilinear image.

Example 136 includes the subject matter of Example 135, and furtherincluding calibrating, by the computing device, a fisheye camera tocapture the fisheye image.

Example 137 includes the subject matter of any of Examples 135 and 136,and wherein determining the region of interest comprises utilizing anoptical flow algorithm to identify a moving object, the region ofinterest corresponding with an image region including the moving object.

Example 138 includes the subject matter of any of Examples 135-137, andwherein determining the region of interest comprises performing objectdetection on the fisheye image to identify an object of interest, theregion of interest corresponding with the identified object of interest.

Example 139 includes the subject matter of any of Examples 135-138, andfurther including projecting, by the computing device, the fisheye imageonto a virtual hemispherical display surface to generate a hemisphericalimage, and projecting, by the computing device, the hemispherical imageonto a plane to generate a perspectively distorted rectilinear image,wherein determining the region of interest comprises performing objectclassification on the perspectively distorted rectilinear image.

Example 140 includes the subject matter of any of Examples 135-139, andwherein performing object classification comprises retrieving an objectprofile from a memory of the computing device; and comparing an objectdetected in the rectilinear image to the retrieved object profile.

Example 141 includes a computing device comprising a processor; and amemory having stored therein a plurality of instructions that whenexecuted by the processor cause the computing device to perform themethod of any of Examples 135-140.

Example 142 includes one or more machine-readable storage mediacomprising a plurality of instructions stored thereon that, in responseto being executed, result in a computing device performing the method ofany of Examples 135-140.

Example 143 includes a computing device for classifying objects in afisheye image, the computing device comprising means for performing themethod of any of Examples 135-140.

1. One or more machine-readable media comprising a plurality of instructions stored thereon that, when executed, causes a computing device to: capture, with a first fisheye camera of the computing device, a first fisheye image including a first scene; capture, with a second fisheye camera of the computing device, a second fisheye image including a second scene overlapping the first scene at an overlapping region; generate a combined image of the first fisheye image and the second fisheye image; perform object classification on a region of interest of the at least one of the first fisheye image or the second fisheye image to classify an object within the region of interest; and generate a portion of a bowl-shaped image based on the combined image.
 2. The one or more computer-readable media of claim 1, wherein the plurality of instructions further cause the computing device to: determine an overlapping image region of the first fisheye image and the second fisheye image; determine movement of objects located in the overlapping image region relative to the first and second fisheye camera based on a plurality of fisheye images generated by the first and second fisheye cameras; identify a moving object in the overlapping image region based on the corresponding relative movement of the objects; and generate an image of the overlapping image region based on the identified moving object.
 3. The one or more computer-readable media of claim 2, wherein to determine the overlapping image region comprises to: perform feature detection on the first and second fisheye images to detect one or more features located in the overlapping region; and perform feature matching on the detected features to determine one or more matched features of the first fisheye image and the second fisheye image.
 4. The one or more computer-readable media of claim 2, wherein to determine the overlapping image region comprises to determine an overlapping image region based on a geometry of a camera system including the first and second fisheye cameras.
 5. The one or more computer-readable media of claim 2, wherein to determine the movement of objects comprises to apply a gradient constraint equation to the plurality of fisheye images.
 6. The one or more computer-readable media of claim 5, wherein to apply the gradient constraint equation comprises to: determine a space derivative of intensity of pixels of the overlapping image region; determine a time derivative of intensity of pixels of the overlapping image region for the plurality of fisheye images; and ensure that ∇I(x,t)u+I_(t)(x,t)=0, wherein ∇I(x,t) is the space derivative, u is a two-dimensional velocity of the plurality of fisheye images, and I_(t)(x,t) is the time derivative.
 7. The one or more computer-readable media of claim 2, wherein to generate the image of the overlapping image region comprises to generate an image that includes the identified moving object from at least one of the first fisheye image or the second fisheye image.
 8. The one or more computer-readable media of claim 2, wherein to generate the image of the overlapping image region comprises to generate an image that includes the identified moving object in an estimated location in the image based on corresponding locations of the identified moving object in the first fisheye image and the second fisheye image.
 9. The one or more computer-readable media of claim 1, wherein the plurality of instructions further cause the computing device to: determine the region of interest in one of the first fisheye image or the second fisheye image; project the region of interest onto a virtual hemispherical display surface to generate a hemispherical image region; project the hemispherical image region onto a plane perpendicular to a point in the hemispherical image region to generate a rectilinear image; and perform object classification on the rectilinear image.
 10. The one or more computer-readable media of claim 9 wherein the plurality of instructions further cause the computing device to calibrate at least one of the first fisheye camera or the second fisheye camera to capture fisheye images.
 11. The one or more computer-readable media of claim 9, wherein to determine the region of interest comprises to utilize an optical flow algorithm to identify a moving object, the region of interest corresponding with an image region including the moving object.
 12. The one or more computer-readable media of claim 9, wherein to determine the region of interest comprises to perform object detection on at least one of the first fisheye image or the second fisheye image to identify an object of interest, the region of interest corresponding with the identified object of interest.
 13. The one or more computer-readable media of claim 9, wherein the plurality of instructions further causes the computing device to: project the one of the first or second fisheye image onto a virtual hemispherical display surface to generate a hemispherical image, and project the hemispherical image onto a plane to generate a perspectively distorted rectilinear image, wherein to determine the region of interest comprises to perform object classification on the perspectively distorted rectilinear image.
 14. The one or more computer-readable media of claim 9, wherein to perform object classification comprises to: retrieve an object profile from a memory of the computing device; and compare an object detected in the rectilinear image to the retrieved object profile.
 15. The one or more computer-readable media of claim 1, wherein the plurality of instructions further cause the computing device to: determine a major semi-axis and a minor semi-axis of a vertical slice of a virtual bowl-shaped projection surface to be generated; determine, for each horizontal slice of the virtual bowl-shaped projection surface, a first semi-axis of the corresponding horizontal slice based on the major semi-axis and the minor semi-axis of the vertical slice; determine, for each horizontal slice of the virtual bowl-shaped projection surface, a second semi-axis of the corresponding horizontal slice perpendicular to the first semi-axis based on an adaptive ratio between the first semi-axis and the second semi-axis of the corresponding horizontal slice, wherein the adaptive ratio for the corresponding horizontal slice is different from each adaptive ratio associated with the other horizontal slices; and generate the virtual bowl-shaped projection surface based on a combination of the horizontal slices.
 16. The one or more computer-readable media of claim 15, wherein the plurality of instructions further cause the computing device to smooth a section of the generated virtual bowl-shaped projection surface adjacent a ground plane of the virtual bowl-shaped projection surface.
 17. The one or more computer-readable media of claim 15, wherein the computing device comprises an in-vehicle computing system of a vehicle, and wherein the plurality of instructions further cause the computing device to: determine a horizontal offset of the corresponding horizontal slice, wherein the horizontal offset is less than a length of the vehicle; and wherein to determine the first semi-axis of the corresponding horizontal slice comprises to determine, for each horizontal slice, the first semi-axis of the corresponding horizontal slice based on the horizontal offset.
 18. The one or more computer-readable media of claim 17, wherein the horizontal offset is greater than half the length of the vehicle.
 19. The one or more computer-readable media of claim 17, wherein to determine the first semi-axis of the corresponding horizontal slice comprises to determine, for each horizontal slice, the first semi-axis of the corresponding horizontal slice as ${{aHorizontal} = {{aOffset} + {{aVertical}*\sqrt{1 - \left( \frac{z - {bVertical}}{bVertical} \right)^{2}}}}},$ wherein aHorizontal is a length of the first semi-axis, aOffset is the horizontal offset, aVertical is a length of the major semi-axis of the vertical slice, bVertical is a length of the minor semi-axis of the vertical slice, and z is a height of the corresponding horizontal slice above a ground plane of the virtual bowl-shaped projection surface.
 20. The one or more computer-readable media of claim 15, wherein the computing device comprises an in-vehicle computing system of a vehicle, and wherein the adaptive ratio changes linearly between adjacent horizontal slices, the adaptive ratio is equal to one for a top horizontal slice, and the adaptive ratio is a ratio of a length of the vehicle and a width of the vehicle for a bottom horizontal slice.
 21. The one or more computer-readable media of claim 15, wherein a top horizontal slice of the virtual bowl-shaped projection surface has a circular geometric shape and a bottom horizontal slice of the virtual bowl-shaped projection surface has a non-circular elliptical geometric shape.
 22. The one or more computer-readable media of claim 15, wherein to determine the second semi-axis of the corresponding horizontal slice comprises to determine, for each horizontal slice, the second semi-axis of the corresponding horizontal slice as a ratio of the first semi-axis of the corresponding horizontal slice and the adaptive ratio for the corresponding horizontal slice.
 23. The one or more computer-readable media of claim 15, wherein the plurality of instructions further cause the computing device to: project an image onto the generated virtual bowl-shaped projection surface to generate the portion of the bowl-shaped image; and display the portion of the bowl-shaped image on a display of the computing device.
 24. The one or more computer-readable media of claim 23, wherein to display the portion of the bowl-shaped image comprises to display a segment of the bowl-shaped image corresponding with a field of view of a virtual camera of a vehicle. 